Health Information Quality of Online Newspaper Articles in Korea

Article information

J Korean Acad Fam Med. 2024;.kjfm.23.0197

Publication date (electronic) : 2024 March 7

doi : https://doi.org/10.4082/kjfm.23.0197

Hangyeol Lee¹, Seung-Won Oh^2,3,

¹Department of Family Medicine, Seoul National University Hospital, Seoul, Korea

²Department of Family Medicine, Healthcare System Gangnam Center, Seoul National University Hospital, Seoul, Korea

³Department of Family Medicine, Seoul National University College of Medicine, Seoul, Korea

^*Corresponding Author: Seung-Won Oh Tel: +82-2-2112-5643, Fax: +82-2-2112-5635, E-mail: sw.oh@snu.ac.kr

Received 2023 September 18; Revised 2023 November 20; Accepted 2023 November 24.

Abstract

INTRODUCTION

Since the birth of the internet, one issue that has received continuous attention is how to obtain health information online [1]. In the past, information that first appeared in the scientific literature was redistributed through the mass media [2], and people relied solely on traditional media and advice from a physician for their health information. Today, however, people seek information online before consulting a physician [3]. Advances in digital media technology have made the internet more accessible, creating additional opportunities to search for health information that was not readily available before, regardless of its reliability or accuracy. The internet has significantly changed how people obtain health information [4].

Experts believed that information consumers would value the same things in evaluating the quality of online information as they did in traditional print publications: authorship, sources, disclosure of conflicts of interest, and timeliness [1]. However, owing to poor access to complete information and the need for high levels of reading comprehension [5], readability and design were also considered important factors [6]. This underscores the role of primary healthcare providers as evaluators and trusted providers of health information in the face of a large information gap between providers and consumers [7].

As can be seen, many factors contribute to the quality of online health information. Moreover, the accuracy and reliability of health information varies considerably and can be misleading [8,9]. The same is true for press releases reporting medical research findings [10-13]. Health information is also considered a social determinant of health because it can have negative consequences if it is incomplete or inaccurate [14].

Multifaceted tools such as HONcode and DISCERN have been developed to assess, and ultimately improve, the quality of health information [15-17]. Evaluation tools have also been developed in Korea [18]. As a result, several studies have evaluated online health information on specific topics, allowing for comparability. However, general-purpose tools that can evaluate various types of online sources are generally rather complex, require a long time to evaluate, and have not proven reliable [19]. The Health Information Quality Assessment Tool (HIQUAL) is the only validated tool to assess the quality of information in online newspaper articles [20,21], but not one study has evaluated the quality of health information in online newspaper articles using this tool.

Therefore, this study seeks to assess the quality of online health-related articles published and distributed through a representative web portal in Korea using a validated tool.

METHODS

2. Selection and Categorization of Articles

The list of collected health articles was screened by a single evaluator (H.L.) by reviewing the titles and text according to specific criteria (Figure 1). Specifically, articles were excluded if (1) they did not contain health information such as policy trends or personnel transfers; (2) they contained health information but did not convey relatively new information; (3) they conveyed relatively current health information but did not meet the classification of health risk factors, diagnosis, treatment, and so forth (e.g., disease epidemiology, new drug release); and (4) they were published in different newspapers, but a significant portion of the article text overlapped, and the evaluation results were the same. The relative recency of an article was judged differently depending on the state of knowledge on the topic. For guidelines, the most up-to-date version was used as the standard, and for individual studies, publication within a period of at least 3 years was used as the standard, with a certain degree of adaptability depending on the topic and content. Of the 712 articles collected, 179 were eligible for analysis. The articles were categorized into three groups based on their content: risk factors, diagnostics, and interventions.

Figure. 1.

Schematic overview of the selection process for articles in the online newspaper.

The selected and categorized articles were independently assessed by two clinicians (H.L., S.W.O.) experienced in assessing the quality of newspaper articles. If the reviewed articles were deemed inappropriate for inclusion in the study through discussion after the evaluation, they were excluded from the study. In cases of disagreement, the scores were reconciled through discussion to finalize the results. Finally, 116 articles were analyzed (Supplement 1).

3. Evaluation Tool

The information quality of each online newspaper article was assessed using the HIQUAL, which has demonstrated reliability and validity for assessing the quality of online newspaper articles. The HIQUAL consists of 10 items in five domains: “reliability,” “usefulness,” “understandability,” “sufficiency,” and “transparency” (Figure 2, Supplement 2). Each item is scored 0/1 or NA if not applicable.

Figure. 2.

Health Information Quality Assessment Tool for newspaper articles (English translated version). NA, not applicable.

To determine the extent to which each item in the evaluation tool was satisfied in the articles studied, we expressed the number of articles for which the item was satisfied, and the number of articles evaluated with that item, in other words, 116-NA in this study, as fractions and percentages. The total score, which represents the quality of the information in each article, is calculated by simply adding up the scores of the individual items. However, if any items are rated NA, the total score is calculated out of 10 by dividing the value added in the same way by the actual number of items rated, in other words, 10-NA, and then multiplying it by 10. In this case, the total score may have a decimal point. The same method was used to calculate the domain score. On the basis of the total score, we categorized the level of recommendation as “not recommended” for a score of 6 or less, “recommended” for a score of greater than 6 and up to 8, and “highly recommended” for a score of greater than 8 [21].

RESULTS

The final quality scores for the 116 selected articles are shown in Table 1. Of the 10 items that make up HIQUAL, “understandability” had the highest item-level satisfaction rate, at 98.3% (114/116). Item 5, “usefulness,” which assesses the reader’s ability to use the information, and item 1 of “reliability,” which examines whether the information is medically valid and based on scientific evidence, had similarly high levels of satisfaction, at 92.2% (107/116). In contrast, item 7 of “sufficiency,” which assesses whether the benefits and harms of the information are presented together, and item 9 of “sufficiency,” which examines whether the costs associated with the information are explained, showed low item-level meet rates of 28.6% (10/35) and 30.0% (9/30), respectively. In addition, items 2 and 3 of “reliability,” which examine whether the source of the information is provided, the timing of the information, and whether the information is current, also showed low levels of satisfaction at 56.9% (66/116) and 59.5% (69/116), respectively. The overall quality rating score indicated that the quality of online newspaper articles was generally acceptable (mean±standard deviation: 7.52±2.00). The number of articles that could be highly recommended was 47 (Table 2).

Table 1.

Quality scores of the articles evaluated by Health Information Quality Assessment Tool

Table 2.

Number of articles according to quality score level (N=116)

We divided the 116 articles analyzed into three categories: 56 articles fell into the category of health risk factors (48.3%); seven articles fell into the category of diagnosis (6.0%); and 53 articles fell into the category of interventions (45.7%). The mean and standard deviations of the total score of the quality assessment were 8.23±1.80, 6.97±2.17, and 6.84±1.95, respectively, and the difference in the total score according to article categorization was statistically significant between the health risk factor and intervention groups (Figure 3).

Figure. 3.

Comparative boxplot of news article quality scores across three categories (N=116). P-values were calculated by Wilcoxon-Mann-Whitney test. NS, not significant. *P=0.00031.

DISCUSSION

Given the importance of determining how often individuals use online health information [22], the need to improve information quality has been emphasized on the assumption that information can have a significant impact on individuals’ health behaviors. To this end, many tools have been developed and applied, including key factors from both the provider and the consumer of information [15-17]. In this context, this study evaluated the quality of articles published in the health information section of Naver News, the most-used portal in Korea, during the first week of May 2023 using the HIQUAL tool. A total of 712 articles were published, of which 116 articles from 27 newspapers were finally selected and analyzed.

The results indicated that the overall quality of health information articles with new content was good. Nearly 40% of the articles were highly recommended, and less than 30% were not recommended, although the criteria for recommendation levels were somewhat arbitrary. This seems to be consistent with the reality that online news portal websites generally provide better-quality information than general websites [23].

If we look at each HIQUAL item, we find, first, that almost all articles satisfied the standard of “understandability,” which assesses whether the information provided is understandable at a layperson level. The same level of high satisfaction was observed with respect to “usefulness,” which reflects whether the reader can accept and use the information in the article, and item 1 of “reliability,” which examines whether the content is reasonable given existing textbook medical knowledge and is presented at an appropriate level of evidence. Taken together, the well-received items suggest that evidence-based knowledge is selected and presented in a way that readers can understand and use. This differs from studies evaluating online health information for pediatric epilepsy, preterm infants, and hand osteoarthritis, which reported low levels of “trustworthiness,” “quality,” and “understandability.” This divergence may be due to differences in the categories of information studied and in the evaluation tools [23-25].

Notably, less than one-third of the articles assessed for sufficiency stated the benefits, risks, and costs of the information provided, and that a significant number of articles contained errors or exaggerations. In particular, the statistical significance of the difference in scores between risk factor and intervention articles confirms that gaps identified in previous studies remain and that newly introduced interventions have an incentive to emphasize their benefits and effectiveness, as does the fact that all five articles that were not recommended fell into the intervention category [11-13]. Furthermore, the finding that nearly half of the articles did not clearly indicate the source and timing of the information is consistent with the results of an evaluation of online information on colorectal cancer in Korea and indicates that further efforts are needed to improve the quality of information [26].

We tested whether scores differed based on article classification in each of the five domains for all articles evaluated, and found that only the difference in scores for “transparency” was statistically significant (Supplement 3). Therefore, the difference in scoring on “transparency” may have contributed to the difference in total score.

To our knowledge, this study is the first to use a validated tool for online health articles in Korea, but it has several limitations. First, this study was conducted by selecting a specific, arbitrarily short time period. To eliminate the bias caused by selecting only articles published in a specific time period, all published articles were screened according to predefined systematic criteria, but the bias may have remained. We believe that screening and selecting published articles by random sampling or systematic clustering may be one way to eliminate this bias. Second, this study focused on targeted articles with relatively new medical content. Many articles about diseases or health-related information of high interest to a population contain no new findings, so numerous articles were excluded from the study. One additional limitation is that the quality of health information on common diseases was not evaluated in a sufficient number of articles. On the other hand, one strength of the study is that the evaluation category was limited to articles containing new information, which is known to be more likely to distort information, in order to identify specific points for improvement. Third, a high proportion of subjects rated NA on items in the “sufficiency” domain. We believe this is due to the inclusion of a large number of articles on health risk factors, for which it is not appropriate to evaluate all health-related articles published through the portal to determine benefits, harms, alternatives, and costs. Therefore, articles dealing with interventions or diagnostics to which all HIQUAL items are applicable should be the focus of future research. Fourth, the number of articles in the diagnosis category was too small to conduct a statistical analysis with high power. As is well known, it takes a considerable amount of time to develop new diagnostics in the medical field; nevertheless, a study with articles published over a longer period of time would be required to evaluate and discuss in depth all areas of the tool in the diagnosis category. Lastly, this study presents results from the perspective of two healthcare professionals. However, because the “understandability” items are intended to be scored from a layperson’s perspective, some form of proxy estimation may have occurred, resulting in an overestimation of the “comprehensibility” scores. While a full correction is unlikely, we believe that some correction may have been possible with a larger number of raters. Nevertheless, this study’s value consists in the following: it provides specific and objective data by classifying and rating the quality of online newspaper articles published on the most popular portal site in Korea using a standardized rating tool with demonstrated validity and reliability by two or more knowledgeable clinicians.

The overall good quality of the articles combined with the lack of improvement in terms of credibility and sufficiency suggests that our understanding of how to assess the quality and credibility of online health information remains limited. A recent meta-analysis study also reported that the quality of health information has a greater impact on health behavior intentions than on actual health behaviors, and that differences in individuals’ information use related to the quality of health information are minimal [27].

Therefore, it may be that the provider of online information and the surrounding support system available to the consumer, such as health professionals [28] or technological tools [29], are more important than the individual consumer. Since the time-consuming application of quality assessment tools has been identified as a blind spot [19], the application of technology to facilitate the two-way effort of improving health literacy at the individual level while simultaneously improving the quality of information could have many benefits [30].

In the future, efficient and ongoing evaluation will be needed to generate representative findings from a larger number of articles with a longer follow-up period, and to more closely analyze the factors necessary to improve the quality of online health information.

SUPPLEMENTARY MATERIALS

Article information Continued