These sources contain the original data and analysis from research studies. No outside evaluation or interpretation is provided. An example of a primary literature source is a peer-reviewed research article. Other primary sources include preprints, dissertations, technical reports and conference proceedings.

These sources provide analysis, synthesis, interpretation and evaluation of primary works. A narrative review article is an example of a secondary source.

The Cochrane Collaboration defines a systematic review as "a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research and to collect and analyze data from the studies that are included in the review." While Cochrane systematic reviews are considered by many to be the gold standard for this type of study, there are several other sources for systematic reviews in behavioral health including:

• Agency for Healthcare Research and Quality (EPC)
• Campbell Collaboration
• NHS Centre for Reviews and Dissemination

When searching for systematic reviews, take note of these questions for evaluating quality:

• Is there a clear statement of the question being addressed?
• Does the review include a thorough description of the search strategy used to locate relevant studies?
• Does the review include a clear discussion of the methods used to both select and evaluate studies?
• Is there an adequate explanation for how the results of the studies were combined?
• Are the conclusions adequately supported by the data that is cited?

A meta-analysis uses statistical methods to combine data from individual, independent studies in order to integrate the findings and synthesize the results. A meta-analysis often evaluates clinical trials to offer guidance on the effectiveness of different therapeutic options. Meta-analytic studies may also cover health care policy issues. Systematic reviews often use meta-analysis to examine differences in treatment effects across multiple studies.

Here is an example of a Cochrane systematic review on behavioral therapy for smoking cessation that uses meta-analysis.

Practice guidelines summarize and refine information on treatment, screening or prevention into practical, focused summaries that reflect current recommended practice. There is one very important point to keep in mind about practice guidelines: they are not all created equal! Some guidelines are based on systematic reviews of the literature. Other guidelines are developed by expert consensus. Different groups, including government agencies, professional societies, governing boards and non-profits, develop these guidelines. As you might expect, evidence-based, systematically researched treatment guidelines, are considered key evidence-based sources. Guidelines can be found in a number of places, including the web sites of a number of associations.

Some sources of practice guidelines in behavioral health include:

• National Guideline Clearinghouse (AHRQ)
• United States Preventive Services Task Force
• National Institute for Clinical Excellence (NICE)
• The Community Guide (CDC)
• Veteran's Affairs/Department of Defense
• American Psychiatric Association

Structured abstracts: some evidence-based resources offer expert commentaries in the form of structured abstracts. One example is the journal Evidence Based Mental Health. Articles included in this journal and others in the "Evidence-based" series of publications are first selected for their quality and practical relevance. Structured abstracts provide added value by summarizing article findings and discussing specific criteria used to assess the quality, validity and practical relevance of the individual research studies.

Most sources of structured abstracts are available by subscription only. For an example of a structured abstract, follow the link for "Editor's choice" on the web site for Evidence Based Mental Health.

Textbooks, if rigorously peer-reviewed and regularly updated, provide useful background information in a concise, easy-to-use format. Textbooks relevant to behavioral practice are increasingly available in electronic formats. Some electronic texts are simply online equivalents of their print counterparts. More and more, however, publishers are offering valuable additions to electronic texts, such as links to PubMed citations, practice guidelines, drug information and news headlines. Most e-texts are available by subscription only, with eMedicine being a notable exception.

Examples of some of the best e-textbooks for behavioral health searching include:

• Clinical Evidence
• ACP Pier
• MD Consult
• DynaMed

One other important concept to understand involves the types of studies in the primary literature that may be used to answer different categories of questions. There is a hierarchy of research evidence (also called the evidence pyramid) that illustrates how knowledge progresses throughout the research process for a treatment question. The base of the pyramid represents initial testing of ideas with animals or in laboratory settings. As you move up the pyramid, ideas are refined and tested in humans. The amount of research literature decreases as you move up the pyramid. However, the evidence at the top of the pyramid is increasingly relevant to answer practical questions about treatments.

Keep in mind that the best research design depends upon the question being asked. Consequently, there is not just one hierarchy of evidence, but rather a different hierarchy for each kind of question. The UCSF medical library provides a very helpful narrated tutorial describing the concept of the hierarchy of evidence as well as several research designs.

Source: Adapted from UNC Chapel Hill Health Sciences Library

The type of study you would look for in the primary literature varies based on the nature of the question being asked. The table below summarizes thinking on the best research study designs corresponding to common categories of clinical questions.

A number of different models attempt to organize and categorize the complex world of evidence-based information resources. One approach uses the concept of background versus foreground questions, as described in the Users' Guides to the Medical Literature.

Background questions often arise when seeing a new patient or facing an unknown clinical condition. In these situations, there is a need for very general information about a disease or disorder. Textbooks and review sources such as UpToDate are examples of evidence-based resources for answering background questions.

In reality, most clinical situations are multi-faceted and complex, resulting in what is known as foreground questions. There are many more search tools for finding relevant information to answer these types of questions. One way to classify these different tools is to talk about "filtered" or "unfiltered" resources. In addition, there is another type of search tool known as a metasearch engine. These resources try to provide "one-stop shopping" for both filtered and unfiltered information.

"Filtered" resources save considerable time and effort when you are looking for the best information to manage and treat patients. In some cases, primary studies are pre-screened and selected for quality and clinical relevance.

One example of an evidence-based, filtered resource is ACP Journal Club, which provides enhanced abstracts of selected studies from over 100 core journals. A very different approach is taken by BMJ's Clinical Evidence, which identifies specific clinical questions. Answers are provided in the form of concise summaries based on systematic literature reviews. Treatment guidelines are another example of filtered resources.

When trying to answer foreground questions, it if often necessary to search the primary literature for original research articles. In behavioral health, these studies are found in a number of literature databases, including Medline (PubMed), PsycInfo, CINAHL and EMBASE.

This type of search can be quite a challenge. Besides needing to locate and master multiple sources, many of the articles included in these databases are not evidence-based. Special search strategies and techniques are needed to filter through the many references to find patient-centered, systematically researched studies.

One way of trying to tame information overload is to use metasearch engines, such as TRIP or SUMSearch. A metasearch engine queries multiple sources. The results are either provided as a single list or displayed according to the original source. Since they draw from many different resources, these tools are useful for answering both background and foreground questions.

Once you have developed a well-defined background or foreground question, you will need to decide where to search for relevant information. With so many options available, it is certainly not always clear where to begin. One helpful approach is to look at a modified version of the "5S" pyramid model of evidence-based information services proposed by Brian Haynes (Haynes RB. Of studies, syntheses, synopses, summaries, and systems: The "5S" evolution of information services for evidence-based healthcare decisions. Evid Based Med. 2006; 11:162-4)

Ideally, patient information in a clinical support system (such as an electronic medical record) would be linked to best evidence or best practices that directly address a patient's circumstances. Unfortunately, having that type of system in place is rare. Instead, you may want to look to a variety of "filtered" sources in the middle of the pyramid, such as systematic reviews, evidence-based summaries, guidelines and textbooks.

If you cannot locate relevant evidence in any of these filtered sources, you will need to search for original journal articles in "unfiltered" databases such as PubMed or PsycInfo which will be covered in more detail in the next section of this module.

In addition to the examples provided in section I of this module, the "Resources" section of the Evidence-Based Behavioral Practice website provides an extensive, annotated list of sources to use when searching for evidence.

The "Resources" section of the Evidence-Based Behavioral Practice website provides a complete, annotated list of sources to use when searching for evidence.

This page lists three main categories of information resources:

• Sources of secondary, synthesized evidence
• Resources to locate primary research literature
• Practice guidelines and empirically supported practices

Another way to help sort through this long list of choices is to start with the type of question and think about the kind of evidence needed. Click on the buttons on the right to see some sample questions and suggested resources for each type of evidence. This information should help guide you in understanding some of the best places to conduct your evidence search.

Keywords are all the same, right? Not true! In the world of clinical and research literature searching, there are two distinctive approaches to locating subjects or topics. True keyword searching allows an individual to select their own words or phrases to describe important concept(s). A thesaurus search, on the other hand, uses a consistent list of vocabulary choices for standardization and searching efficiency. Thesaurus searching is also known as controlled vocabulary searching.

MeSH, or the Medical Subject Headings system used with the Medline (PubMed) database is a good examples of a thesaurus. Each article in the Medline database is assigned ten to twenty MeSH index terms to describe the content of the article. Using MeSH subject terms for searching provides greater control and precision.

PsycInfo also provides a powerful thesaurus of standardized subject terms to aid in searching consistency and comprehensiveness. The PsycINFO Thesaurus consists of over 8200 records derived from the Thesaurus of Psychological Index Terms.

There are two other literature databases containing behavioral practice articles that use thesaurus searching: CINAHL (Cumulative Index to Nursing and Allied Health Literature) and EMBASE. Both of these services are available by subscription only.

Boolean operators are used for combining concepts or establishing relationships when searching. The three commonly used Boolean operators are AND, OR and NOT. Using these terms can broaden or narrow your results. Parentheses can be used for more complex Boolean searches.

AND is used to join words or phrases when you want both (or all) of the search terms to be included in your results.

OR is used for finding synonymous terms or phrases. Using this operator yields more results than if you searched for only one concept.

NOT is used to exclude a word or phrase from your search results. It is wise to use this Boolean term sparingly, though, since you may end up eliminating relevant results.

Many search systems allow the use of truncation to retrieve variations on spelling or word endings. A truncation symbol, most commonly the asterisk (*), stands for one or more unspecified characters at the end of a word. By using a truncation symbol, you allow a search engine to look for related, relevant words you do not have to type in yourself.

Many search engines are very literal in how they interpret your search. They will only match the exact keyword or phrase that you use. To enhance your results and do a more comprehensive search, you may want to make use of truncation. For instance, if you enter the word smoke without a truncation symbol, you would miss references that use the words smoking, smokes or smoked. In order to retrieve all of these terms, you would simply type in smok*

An important note - using truncation is not necessary for resources that use thesaurus terms, such as Medline (PubMed) or PsycInfo.

A very useful way to focus your search is to use "limit" options to refine your search results. Many databases and search engines allow you to target your search to find particular study types such as randomized controlled trials. Some, such as CINAHL, actually provide an option to limit to a subset of the main database that is considered to be evidence-based. PsycInfo provides some unique options for limiting, including "Classification Codes" (e.g. Neuropsycholgical Assessment) and Intended Audience (e.g. Professional or General Public).

Some of the most common "limits" you can expect to find are:

• Date of publication
• Language
• Publication type (case studies, RCTs, etc)
• Age groups
• Type of materials (books, journals, dissertations, etc.)

For a Medline search using PubMed, one particularly helpful option is using the built-in filters provided through the Clinical Queries feature. These specialized filters, according to the PubMed help file, "limit retrieval to citations to articles reporting research conducted with specific methodologies, including those that report applied clinical research."

The main PubMed search options on the clinical query page include:

• Etiology
• Diagnosis
• Therapy
• Prognosis
• Clinical Prediction Guides

Click on each of the links below to see video examples of how to search these sites:

Cochrane

• Cochrane video

PubMed (in two parts)

• PubMed video part 1
• PubMed video part 2

PsycInfo

• PsycInfo video

Citation searching

What exactly is citation searching and why would you use it? It is logical to expand on subject or keyword searching by going backward in time and following the links to related works that are listed in a known article's bibliography.

The other part of citation searching is going forward in time to discover who has cited the original article or an author of interest. This process is very valuable for uncovering the "web" of knowledge or scholarship on your topic.

Several proprietary systems allow citation searching, including ISI Web of Knowledge, Scopus and PsycInfo. Google Scholar, which is freely available, also has a "cited by" option although it is considered incomplete and unreliable.

Hand searching

To round out a systematic search, it may be helpful to manually browse or scan articles in key journals. Hand searching of meeting abstracts or "grey literature" sources may also uncover important works not identified through keyword or citation searching.

The Cochrane Collaboration's training manual for handsearching states that there "appears to be no alternative" if the aim is "complete identification of published reports." Unless the searcher is conducting a true systematic review, however, this step is most likely not necessary.

It is certainly tempting to think you are completely finished with your search after your first attempt at finding the best evidence on your topic. But it is almost always worth taking a few extra minutes to review your results and think about possibilities for revising your original approach.

The ideal evidence-based search engine would allow you to find all the high-quality, relevant information you need in one easy step. In his article on Google Scholar as a source for clinicians, Jim Henderson offers his vision of a model finding aid:

"The ideal tool for finding clinical information would be a fast engine that provides the best hits from scholarly journal literature and clinical resources such as guidelines, perhaps emphasizing sites favoured by physicians in the way that Google emphasizes popular websites for general audiences. Busy clinicians would wish for succinct reviews and for the best evidence, with links to key papers that would be determined as such by the number of times they have been cited, thus balancing popularity with relevance and quality. Features enabling search refinement would be welcome, such as a tool to find related articles by subject or by using links or citations, including more recent articles that cite the retrieved items. Ideally, this engine would provide integrated, powerful access to many sources, including full-text journal literature and textbooks, evidence-based information, information for patients, and drug information, achieving for clinical sources what Google has for the entire public Internet."

—Source: Henderson J. Google scholar: A source for clinicians? CMAJ. 2005;172:1549-1550

Metasearch engines such as TRIP or SUMSearch, (discussed in Part II of this tutorial) aim to provide a single Google-like option for locating the best evidence. There are drawbacks to these search tools, however. One major issue is the inability to link to many of the full-text resources due to restricted access.

In the absence of one exemplary finding aid, the burden is on the individual to make sure that all (or at least most) of the best resources are included in the evidence search.

Beyond making sure you have searched the key resources, you need to be aware of some of the most common searching mistakes. Here are a few to note, along with tips for improving your results:

Not using the thesaurus terms or subject headings when searching for primary literature.

There are four key databases for finding primary studies on behavioral health topics that use "controlled vocabularies"- Medline, PsycInfo, CINAHL and Embase. If you do not use these standardized terms, you will either miss relevant articles or retrieve lots of "false drops." Because different information companies offer access to these databases, exactly how you access the thesaurus terms and include them in your search strategy varies. Review the video clips on searching PubMed and PsycInfo in Part IV of this tutorial for examples of how to find and use thesaurus terms.

Using too many search terms

It may be tempting to use all four of the elements of your "PICO" question when you do your search. The best approach, however, is to start by focusing on two or three points so your initial search is not too restrictive. You can always add additional concepts or apply "limits" (such as an age range) to your search strategy if you get too many results.

Using the wrong Boolean operator

Boolean operators seem simple enough. However, in the midst of trying to master different search tools, it is easy to loose sight of how and when to use AND or when it is appropriate to use OR. Review the information on Boolean operators in Part IV of the tutorial if you do experience this problem.

If you need additional help, see this tutorial on Boolean operators from the New York University Libraries.

In most cases, you do not need to track or document your literature search process for an individual clinical case. However, you may want to consider keeping a record of the search process for related questions that may arise at a future time.

In some cases, you may also want to share your experiences with colleagues, either formally in the literature or informally at conferences or meetings. The Cochrane Collaboration's Handbook offers detailed guidance on Searching for Studies, including guidance on "documenting and reporting the search process."

Here is a list of the essential elements needed to track your search:

• Titles of databases searched (e.g., PsycInfo)
• Names of the hosts or systems (e.g., EBSCOhost)
• Date search was run (month, day, year)
• Years covered by the search
• Complete strategy used, including all search terms and the winnowing process (dropping references on the basis of study design, date of publication, sample size, etc.)

We have now completed our journey towards understanding how to search for evidence to support clinical decision-making for behavioral health issues. Good luck with your searching!