The Benefits of Colorectal Surgery Surveys in Australia and New Zealand

Article information

Ann Coloproctol. 2020;36(2):102-111

Publication date (electronic) : 2020 April 30

doi : https://doi.org/10.3393/ac.2019.09.17

Auerilius Erastus Ricardo Hamilton^1,2, Amelia Alice Lin^1,2, Christopher John Young^,1,2

¹Department of Colorectal Surgery, Royal Prince Alfred Hospital, Camperdown, New South Wales, Australia

²The University of Sydney, Central Clinical School, Sydney, New South Wales, Australia

Correspondence to: Christopher J. Young, MS FRACS FACS FASCRS Department of Colorectal Surgery, Royal Prince Alfred Hospital, Camperdown, New South Wales, Australia Tel: +61-2-9515-7576, Fax: +61-2-9515-1806 E-mail: cyoungnsw@aol.com

Received 2019 July 18; Accepted 2019 September 17.

Abstract

INTRODUCTION

Some academic surgeons feel that the number of surveys sanctioned by the Colorectal Surgery Society of Australia and New Zealand (CSSANZ) should be culled due to high nuisance effects and limited academic value of the surveys. We aim to present an opposing viewpoint in defence of surveys. Some surgeons within the CSSANZ argue that surveys do not sufficiently meet the research standards mandated for trainees. The Research Support Committee (RSC) within the CSSANZ oversees all matters pertaining to surveys. The RSC approves 4 to 6 surveys per year, although this number may increase to include surveys not requiring reminders or those sent from affiliate organisations. Researchers often acknowledge their collaboration and cooperation with frequent partners including General Surgeons Australia, New Zealand Association of General Surgeons, the Gastroenterology Society of Australia, and other medical or social coloproctology-focused groups.

Anecdotally, most CSSANZ surveys have had high response rates (RRs) in comparison to RRs for general surgical surveys, and CSSANZ surveys largely result in publications. Surveys with opaque conclusions due to poor responses struggle to get published unless they are well constructed with interesting topics. Four or more researchers often draft surveys in collaboration before submission to CSSANZ for consideration. The logistical processes of submitting and completing surveys are achievable, consistent, and clearly known. Submissions undergo rigorous screening and scrutiny by RSC officers, often with 9 to 13 passes in the drafting process, before progressing to general committee review. Successfully screened surveys that are circulated traditionally return quick, helpful, and much-appreciated responses by assiduous members. In recent years, RSC officers have anecdotally noticed a reduction in RR in cycles wherein higher numbers of surveys are distributed, or in cases where surveys are spearheaded by non-members (regardless of investigators’ level of training). Surveys tend to be more successful if they are sponsored by CSSANZ members (especially for non-member applications), if they are well constructed, if they avoid repetitive topics, and in correspondence to online versus paper copies. Between the 2 countries of Australia and New Zealand, surveys distributed by post present great challenges, which are slightly mitigated when researchers use prestamped, self-addressed return envelopes. Some responding surgeons have kindly scanned paper surveys and returned them by email.

Following completion of their research, investigators face the arduous task of securing publication in reputable journals. Anecdotally, journal houses in our region disfavour publishing survey-related research, preferring instead to change survey-based articles to Perspective papers of 800–1,000 words. Researchers are then driven to publish externally in other international journals with higher impact factors with larger readership numbers. Hence, opportunities are lost to locally showcase homegrown research.

Surveys struggle to gain acceptance in the discourse on general surgery, and not surprisingly, the same hurdles are faced in colorectal surgery. Surveys face the challenge of overcoming many biases both intrinsic and extrinsic. Intrinsic biases in cross-sectional studies can include selection bias, nonresponse bias, information biases such as recall bias and detection bias, and confounding bias. Extrinsic biases can come from individual readers, editorial organisations, journal houses, and speciality group prejudices. We hope that the present study reduces some of the extrinsic biases toward colorectal surveys in Australia and New Zealand.

METHODS

This study aims to assess the outcomes and benefits of CSSANZ-distributed surveys over a 9-year period. It addresses several questions, as follows. (1) In typical surveys, what topics are being investigated and which question types are being asked? (2) What rate of CSSANZ surveys formally progresses to positive literary outcomes in publications and/or dissertations for higher degrees? (3) How does the publication rate for these CSSANZ surveys compare to the publication rate for submitted Royal Australasian College of Surgeons research abstracts?

Between 1st January 2009 and 31st December 2017, the CSSANZ distributed 38 surveys to its members. Subsequently, all of the surveys were retrospectively analysed as of census date 30th June 2018. The majority of these surveys (92%) were spearheaded by CSSANZ members. A few surveys (8%) were looked after by other affiliate groups—namely, the Lowry Cancer Research group of the University of New South Wales, Crohn’s and Colitis Australia, and the Gastroenterology and Liver Services at Concord Hospital NSW of the University of Sydney. Each survey had its own initial ethics approval, and later, the CSSANZ Executive Committee approved a comparative “survey of these surveys.” We reviewed all questionnaires held by the Secretariat and manually counted the number of questions for each. All publications were found via PubMed, MEDLINE, and/or Google Scholar. For our purposes of analysis herein, survey types are categorised into discrete categories of referral pathways, current practices, decision-making, or knowledge and skills assessment. For consensus, our authors independently categorised each survey based on their title wording, questionnaire structure, or conclusions from subsequent publications. Where initial consensus was not met on categorisation, agreement was later finalised by panel discussion. Survey topics are categorised into one of the listed chapters from the Table of Contents from the American Society of Colon and Rectal Surgeons Textbook of Colon and Rectal Surgery [1]. Survey times refer to the estimated minutes required for participants to complete a survey as stated by investigators. Time to publication in months measures the time from survey submission to the Secretariat for distribution to the acceptance date for publication according to the article title page or the electronic-publication date in the citation.

The survey delivery method is listed as either “paper” (for telephone or postal mail) or “electronic” (online platforms or emailed questionnaires) depending on response collection as opposed to questionnaire-distribution. Those with online questionnaires used platforms including KeySurvey (Braintree, MA, USA); SurveyMonkey (SurveyMonkey, San Mateo, CA, USA); Google Docs (Google, Mountain View, CA, USA); or Vista (Vanguard Software Corp., Cary, NC, USA). Links to questionnaires were sent by email or by advertisements in relevant societies’ newsletters or journals. Commonly, after a questionnaire was dispersed, 1 or (on rare occasion) 2 reminders were issued as follow-up at intervals of 2 weeks apart. After the passage of 2 months, many investigators ceased to accept survey responses.

RRs from CSSANZ members were calculated with respondent numbers reported by investigators functioning as the numerator and questionnaire-distribution numbers reported by the Secretariat functioning as the denominator. RRs from non-members were chart listed, but not included in our analysis. An RR was “not defined” for surveys generically sent to non-CSSANZ members of large organisations where only the numerator of respondent numbers was known. Due to missing data from unpublished surveys for some of the defined elements above, complete analysis of RR for all 38 surveys was not possible.

The authors individually scored all surveys with the newly developed yet unvalidated qualitative Young’s Survey Score (YSS), shown in Fig. 1, as developed by this paper’s senior author (CJY). The YSS gives a rating according to 5 questions asked of each survey, with scores marked out of 10 for a maximum total of 50 points. The scores examined for each survey include dimensions of the topic’s interest and importance to CSSANZ membership, currency of similar topics in circulation, length of time required for respondents to complete surveys, and ease of answering based on delivery method. The summative score intends to quantify these 5 variables as a novel quality scoring system for surveys in colorectal surgery. For consensus, an average was taken of individual authors’ scores for each survey.

Fig. 1.

Young’s Survey Score developed by the senior author.

RRs from unpublished surveys were unknown by the Secretariat or were not obtainable from the primary investigators at the time of this study. Based on published surveys, we listed the topics of each survey according to their topic subject matter, and divided them into high RR (≥50%) or low RR (<49%) by CSSANZ members. As of the study census date, we did not have full details regarding the fate of unpublished surveys. Two surveys are confirmed to have ceased—1 due to poor responses and 1 that did not progress. The other surveys are currently facing the ongoing challenge of finding a suitable journal organisation for publication.

RESULTS

Over the 9-year study period, the CSSANZ annually distributed a mean ± standard deviation (SD) of 4 ± 2 surveys with a mean annual publication rate of 2.2 (55%), as shown in Fig. 2. Regarding the 38 surveys overall, the 3 most common survey topics, as shown in Supplementary Table 1, were rectal cancer decision-making, in 7 publications (16%), preoperative assessment of colorectal patients, in 6 publications (13%), and anal physiology: continence and defaecation, in 5 publications (11%). The collective outcomes of the surveys’ delivery methods, number of questions, number of pages, survey quality, distribution per year, and survey type frequency of use are shown in Table 1.

Fig. 2.

Number of surveys distributed and subsequently published each year by CSSANZ from 2009 to 2017. CSSANZ, Colorectal Surgery Society Australia and New Zealand.

Table 1.

Demographics and survey types of CSSANZ surveys, 2009–2017 (n = 38)

Regarding positive literary outcomes, 20 surveys (53%) were published as of census date 30th June 2018. The survey results are shown to have contributed to 13 higher degrees, comprising 3 PhDs and 10 Masters degrees, as shown in Supplementary Table 2. Fig. 3A [2-21] and Supplementary Table 3, respectively, show a summarised and expanded list of these surveys and their RRs. For published surveys, the mean ± SD of RR from CSSANZ members was 53% ± 18.3%, as depicted in Fig. 3B, and the mean ± SD impact factor was 2.41 ± 1.55 for publishing journals, as shown in Fig. 4A [22-24]. Fig. 4B shows a mean ± SD time to publication for relevant surveys of 31 ± 17 months. Fig. 5 shows a Kaplan-Meier analysis with “publication” being the event of interest. Surveys with a YSS of ≥37/50 were more likely to be accepted for publication (P = 0.027), with a mean publication time of 36.5 months for this subgroup.

Fig. 3.

Summary of CSSANZ surveys 2009–2017 subsequently published. (A) Number of individual surveys sent to candidates and received from respondents. (B) Response rates to each survey project from all respondents and from CSSANZ members. CSSANZ, Colorectal Surgery Society Australia and New Zealand.

Fig. 4.

Summary of CSSANZ surveys 2009–2017 subsequently published. (A) Impact factor of journal of publication for each survey project. (B) Time to publication for each survey project. CSSANZ, Colorectal Surgery Society Australia and New Zealand.

Fig. 5.

Time to publication by Survey Score groups for CSSANZ surveys 2009–2017 subsequently published. CSSANZ, Colorectal Surgery Society Australia and New Zealand; SE, standard error; CI, confidence interval; KM, Kaplan-Meier; HR, hazard ratio.

For the 38 surveys, comparison analyses of the methods of delivery and progression to publication are shown in Tables 2 and 3. The progression of surveys to publication was not influenced by the number of survey questions, number of pages, time required to complete the survey, or the number of surveys distributed in a year. Investigators undertaking a higher degree were more likely (P < 0.001) to use paper-based surveys. The factors used for univariate and multivariate analysis were surveys’ methods of delivery, YSSs, and occurrence in the first or second half of the study period to remove lead-time biases, as shown in Table 4. From these analyses, surveys with a YSS of ≥37/50 were significantly more likely to be accepted for publication. Table 5 shows topics from 20 published surveys and their categorised RR with no obvious correlation between the topic subject matter and the incidence of high or low RRs.

Table 2.

Comparison of CSSANZ surveys 2009–2017 by method of delivery (n = 38)

Table 3.

Demographic comparisons of published and unpublished CSSANZ surveys, 2009–2017 (n = 38)

Table 4.

Univariate categorical analysis and Cox regression analysis of factors affecting time to publication of surveys

Table 5.

Comparison of response rate and topic subject matter

DISCUSSION

This study showed that CSSANZ-distributed surveys had a high RR from members, with the majority resulting in publication and with one third of completed surveys contributing to investigators’ advancement to higher degrees. Investigators doing higher degrees more commonly (P < 0.001) used paper-based questionnaires, probably due to the clerical structure, research support resources, and preference of common research offices wherein many of them matriculated. There was no significant influence on the progression of surveys to publication due to the number of surveys distributed each year, the number of questions, number of pages, method of delivery, or the time required for participants to complete the surveys. This refutes previously held beliefs about disincentives in potential respondents’ desire to participate in surveys among CSSANZ members. There were combinations of surveys types used with the following frequencies: referral pathways, in 1 survey (3%), current practice, in 33 surveys (87%), decision-making, in 17 surveys (45%), and knowledge and skills assessment, in 16 surveys (42%). Through analysis herein, surveys are shown to aid research into areas that are otherwise hard to assess. Moreover, surveys often indicate areas for future research. For example, a questionnaire on diverticulitis circulated by Siddiqui et al. [20] revealed a lack of consensus, and highlights practices that disagree with current treatment guidelines. This meaningful discrepancy certainly needs further explanation via future research. These findings are not discernible from randomized controlled trial (RCTs), which cannot predict opinions. Areas of investigation and question types that are typically addressed in surveys cannot be answered by RCTs. Importantly, surveys tend to lead to or unearth areas where RCTs may be needed for more robust understanding of current topics.

In comparison to regional trends for all research types from all surgical disciplines, CSSANZ surveys are shown to have a high mean publication rate (53%), but a long mean ± SD publication time of 31 ± 17 months. The investigative work of Khajehnoori et al. [25] shows that 26% of abstracts submitted to the Annual Scientific Congress of the Royal Australasian College of Surgeons were subsequently published, with 74% of these accepted ≤24 months from original submission of the abstract.

Of the 38 surveys, 6 surveys (16%) were published in regional journals. Despite reviewing 117 self-administered surveys from 34 high impact factor international journals over 1 year, Bennett et al. [26] found that most journals provided no guidance to authors for reporting survey research. The work asserts that survey-based research needs to be reported transparently so readers and reviewers can critically assess the strengths and weaknesses of the design, methodology, and analysis therein. Currently, there is no quality scoring system for surveys in colorectal surgery. Thus, we propose our YSS for internal assessment of CSSANZ-distributed surveys. In the future, we hope to analyse whether there are correlations between YSSs, rates of responses, and publication.

The work of Edwards et al. [27] gives practical tips for improving survey-based RR data collection in postal and electronic questionnaires, although our study shows that neither method influences progression to publication. The work of Hing et al. [28] suggests that questionnaire RR can be improved by optimising study design and including incentives. Despite assessing survey design, our study is not able to analyse the influence of incentives due to missing information from our cohort. According to Alderman and colleagues, the medical research field uses 3 main types of surveys: (1) epidemiological surveys, (2) surveys on attitudes toward a health service or intervention, and (3) questionnaires assessing knowledge on an issue or topic [29]. In CSSANZ-distributed surveys, our categorisation was into 4 survey types, closely paralleling the above.

We advise future survey investigators to consider the following:

• Dutifully report to the Secretariat all positive literary outcomes including public showcasing at scientific meetings (oral presentations, posters, and/or abstracts), or at workshops (oral presentations).

• Analyse how your survey influences policy changes in your organisation at various levels (surgical units, department-wide, hospital-wide, and/or area/regional health districts).

• Utilise smartphone technologies with applications linked to secure online survey platforms for more efficient and rapid questionnaire delivery, reminder distribution, and response collection. Numerous commercial applications currently exist, and in the future, applications may be custom-developed for the CSSANZ.

• Consider sending up to 4 reminders fortnightly, ideally electronically, as one reminder seems politely insufficient.

Following this study, we will advocate for an increase from 4–6 surveys to 7–8 surveys per year, with similar expectations for positive literary outcomes, as proven herein. Changes in journal submission policies, along with consistent and clear reporting guidelines for colorectal surgery survey research, may help retain publication of future surveys in Australasia. Surveys indicate areas of equipoise where further research may be useful. We advocate for greater discussion within the CSSANZ to use each sanctioned survey as an impetus for greater policy discussion, for more frequent scrutiny of guidelines, and for further research by its membership.

Future investigators are encouraged to clearly state the immediate aims and long-term goals of their surveys. When CSSANZ surveys appear to forecast trends, deviations from guidelines, or insights beyond the scope of opinion polls or practice audits, this shift may lead to increased survey acceptance by colorectal surgeons and to better survey assessment of benefits in clinical practice. All past investigators fully appreciate the support of CSSANZ membership in assisting in projects that stand to help all practitioners in colorectal surgery understand areas for further research. The invaluable amount of time spent, contributions made, and the innate culture of research camaraderie among members facilitates cooperative participation and commendable RRs to these important surveys.

This was a retrospective review with all the intrinsic limitations of nonrandomisation. In addition, our review is limited by the nonresponse bias, which is a weakness of all surveys. As highlighted in this study, the impact factor only represents a score for the popularity of journal readership, and not for research quality. Missing information about respondents’ locations, duration of time spent to collect data, distribution of reminders, and RRs from unpublished surveys prevent a more in-depth analysis of non-RRs.

The present study does not explore the multifactorial factors that influence RRs (negatively or positively). Our data collection was significantly incomplete, and hence, contrary to the plan of study, no analysis was made in areas including time taken by investigators to collect data, number of reminders sent, method of reminder, timing of reminder, recruitment incentives used, geographical location (country, state, and city) of surveyed participants, and age of participants. The small sample size may have accounted for the study’s inability to reach significance in showing the influence on publication rate by the number of surveys distributed each year, the number of questions, number of pages, method of delivery, and participant time required for completion of surveys. This information would have facilitated better assessment of the ways to more effectively deliver surveys to CSSANZ membership, whether pertaining to delivery method (paper versus electronic), the use of incentives, and/or specific geographical challenges, with the common aim of increasing participation by colorectal surgeons. Again, we intuitively forecast that RRs may be improved by a trial of a CSSANZ custom-designed mobile phone application, together with a system of continual professional development points awarded to respondents.

The study also does not review the reasons for surgeons’ deviation from conventional guidelines. Any deviation from standard practice should be a cause for concern, as it either points to better treatment options or to an inadequacy of the guidelines. Whereas the opinions of the expert masses should not be followed blindly to determine policy, deviations from policy by the masses should prompt a prudent review. Robust discourse should take place among our experienced and knowledgeable membership regarding the best ways to customise guidelines to align with existing regional experiences.

Lastly, our study does not assess how the results of completed surveys have influenced further research or have led to policy implementations or changes in respective healthcare centres. This is a matter for future review by CSSANZ membership, as it should specifically investigate how these surveys may have influenced clinical governance and administration throughout the region.

In conclusion, surveys in colorectal surgery can be very useful and should continue to be considered for use by the CSSANZ. Surveys overall benefit the CSSANZ by providing a valuable and productive research methodology for auditing and reviewing the membership’s clinical practices and adherence to national and regional guidelines. Ideally, CSSANZ surveys should address a popular, trendy, or even controversial topic, should specifically or exclusively target proctology colleagues (members and affiliates), and should be of high quality according to YSS assessment. To specify dimensions of high quality, optimal surveys are interesting, important, pertinent to topics not surveyed within the last 2 years, require less than 10 minutes to complete, and are easy to return by any means.

SUPPLEMENTARY MATERIAL

Article information Continued