A descriptive analysis of the replication applied in aquaponic experimental studies

A literature search was performed via SCOPUS for publications between January 2000 and April 2020 that contained the keywords aquaponic and hydroponic. Sixty-one articles were identified that stated a comparison and a form of comparative statistical analysis was performed. These articles were identified via the principle author, year of publication, the system type tested (coupled or decoupled aquaponic; irrigated nutrient solution from a separated RAS), the number of treatments tested, the number of replicates applied to each treatment and the region or country within which the experiment was performed. An experimental comparison context was assigned to each study to identify the requirement for replication. Sixty-one percent (61 %) of all the studies were deemed to have applied no or incorrect replication (no or incorrect replication: 56 % of fully recirculating system studies, 100 % of decoupled system studies, 86 % of irrigated RAS water studies). In terms of the comparison context, 54 % of system comparison studies, 100 % of solution comparison studies and 63 % of plant component associated comparison studies, applied no or incorrect replication. The association between study location and the incidence of no or incorrect replication was also determined (Europe – 71 %, USA – 80 %, South America – 63 %, Australia – 33 %, West Asia – 5 %, South East Asia; including China – 20 % and South Asia – 14 %). An experimental replication decision matrix was developed to assist future aquaponic researchers in determining the application of correct replication and several example research articles were discussed to demonstrate and explain the correct and incorrect application of replication in experimental designs for aquaponic associated research studies. Keywords— Aquaponic, Hydroponic, Aquaculture, RAS, Replication, Statistics, Experimental design.


INTRODUCTION
Aquaponics is a technology that produces both animal (usually fish) and plant products and confers advantages such as water savings, nutrient input savings, lowered environmental impact and an ability for universal location (Lennard, 2017;Ayipio et al., 2019). A proportion of broader aquaculture production is done using Recirculating Aquaculture Systems (RAS), which are analogous to the fish culturing component in an aquaponic context (Lennard, 2017). Vegetable production is partly being done using hydroponic or substrate culture technologies, which are analogous to the plant culturing component in an aquaponic context (Lennard, 2017). Therefore, both fish and vegetables are being produced with existing waterculturing technologies that are analogous to aquaponics. This, along with the intrinsic goal of improving all efficiencies associated with aquaponic production technologies and methodologies, allows a comparative pathway for aquaponics in terms of fish and vegetable production (i.e. fish and plant growth rates may be compared between RAS and aquaponics and between hydroponics and aquaponics to establish the relative efficiencies of the aquaponic technique).
Much aquaponic research is associated with establishing and/or improving the productive potential of the technique, especially via comparison to conventional hydroponic crop production (Ayipio et al., 2019). To measure and establish any perceived improvement, researchers compare the outputs of aquaponics (e.g. plant growth or yield) with either internal aquaponic variables (e.g. fish species cultured, fish stocking density, solution pH, presence/absence of additional nutrients, presence/absence of solution sterilisation, hydraulic parametersflow rate or hydraulic retention time, planting media, etc.) or externally appropriate comparative technologies with known high productive rates for the associated organism (e.g. hydroponics and substrate culture -plants) (Ayipio et al., 2019). Therefore, especially in terms of plant growth and production, a proportion of aquaponic research is associated with experimentally comparing a proposed aquaponic technology variation with a known hydroponic culturing technique which possesses expected plant production outcomes that are generally considered to be industry best practice (Resh, 2013).
A pivotal requirement of comparative research is the correct application of experimental design, which allows the researcher(s) to determine that the comparison they envisage and desire to test may be done in a context where the measured outcomes may be clarified and verified via statistical analysis (Hurlbert, 1984 The aim of this study was to analyse the context of comparison and the associated replication applied to that context, in a randomly selected cohort of peer reviewed, published, scientific articles about aquaponic technology which applied scientific comparisons. The studies were examined, and the context of comparison determined. The replication applied was then examined to determine if it was appropriate for the context of the comparison. The outcomes of the analysis were then used to determine the frequency of inappropriate replication applications within several sub-sets of aquaponic research. The analytical outcomes were then used to develop a decision matrix for researchers in an attempt to improve the application of appropriate replication in experimental designs for aquaponics experimental research that could produce valid, comparative statistical outcomes.

II. MATERIALS AND METHODS
A literature search was performed between the dates 12 th May 2020 and 19 th May 2020, via SCOPUS. Sixty-one articles (excluding reviews and editorials) published in peer reviewed journals were identified between January 2000 and April 2020 that contained the keywords aquaponic and hydroponic, stated a comparison for some form of plant growth or production parameter, stated the inclusion of replication and stated that a form of comparative statistical analysis was performed.
The identified studies were tabulated to include the name of the first author, the year of publication, the system type(s) applied (see below), the number of treatments applied, the number of replications applied per treatment, the region or country within which the experiments were performed and the applied context of comparative analysis.
In terms of the system type, a recirculating aquaponic system was identified as a system that contained fish (in a fish tank), a form of filtration (mechanical, biological or both) and a hydroponic component where the system water recirculated between the fish and plant components perpetually. A hydroponics system irrigated with RAS water was identified as a system not containing fish, whereby a hydroponic component was irrigated with water that originated from a separate RAS that was not part of the experimental design (i.e. the RAS was not connected to the hydroponic component of the experimental design in any hydraulic form or sense, but RAS water was used to fill the sump or nutrient tank of the experimental hydroponic component). A series of descriptive statistics were identified and tabulated to include the total number of studies identified, the number of fully recirculating system designs, the number of decoupled system designs and the number of irrigated RAS water designs, the number of system comparisons, the number of solution comparisons and the number of plant component comparisons, the number and percentage of correctly replicated studies, the number and percentage of incorrectly replicated studies and a breakdown of the same descriptive statistics based on the region or country of location of the research.
The above outlined information was then used to develop an experimental design decision matrix so that future researchers have an available primer for correct experimental design in an aquaponic research context. In addition, several studies were isolated, examined and discussed in terms of the applied experimental design and replication to provide examples of valid and invalid application of replication, to support and illustrate the developed decision matrix. Table 1 outlines all sixty-one identified studies included in the analysis (by primary author), and includes the year of publication, the system type tested (Recirculating or Coupled, Hydroponics Irrigated with RAS Water and Decoupled), the number of treatments tested, the number of replications applied to each treatment, the region or country where the experiments were conducted and the comparison context.

Study Analysis and Descriptive Statistics:
A series of descriptive statistics for the identified studies are presented in Table 2. Of the sixty-one identified studies, fifty-two were identified as testing recirculating (coupled) aquaponic systems, two were identified as testing decoupled aquaponic systems(where fish were present in a connected, hydraulic context) and seven were identified as testing a variation of irrigating water to a hydroponic component that was sourced from a separate, operating RAS.
Forty-six studies were identified as comparing the whole or complete aquaponic systems (i.e. system comparisons), seven studies were identified as comparing nutrient solution(s) (i.e. solution comparisons) and eight were identified as comparing a variation associated with a plant component (i.e. plant component comparisons). Of the forty-six system comparisons, twenty-one (46 %) were identified to have applied correct replication of the treatments tested and twenty-five (54 %) were identified to have applied incorrect replication of the treatments tested.
Of the seven solution comparisons, zero (0 %) were identified to have applied correct replication of the treatments tested and seven (100 %) were identified to have applied incorrect replication of the treatments tested.
Of the eight plant component comparisons, three (37 %) were identified to have applied correct replication of the treatments tested and five (63 %) were identified to have applied incorrect replication of the treatments tested.
Of the fifty-two recirculating (coupled) aquaponic systems studied, twenty-three (44 %) were identified to have applied correct replication of the treatments tested and twenty-nine (56 %) were identified to have applied incorrect replication of the treatments tested. Of the seven hydroponic components irrigated with RAS water and/or hydroponic nutrient solution controls, one (14 %) was identified to have applied correct replication of the treatments tested and six (86 %) were identified to have applied incorrect replication of the treatments tested. Of the two decoupled aquaponic systems studied, zero (0 %) were identified to have applied correct replication of the treatments tested and two (100 %) were identified to have applied incorrect replication of the treatments tested.
Overall, of the sixty-one identified studies, twenty-four (39 %) were identified to have applied correct replication of the treatments tested and thirty-seven (61 %) were identified to have applied incorrect, or no, replication of the treatments tested. Table 2 also contains a regional breakdown of descriptive statistics which includes all of those outlined above, within a regional context. Importantly, this outlines the percentage of studies performed per region that applied incorrect or no  Experimental Replication Decision Matrix: Figure 1 outlines a Decision Matrix to enable the determination of valid replication for the devised experimental comparisons in an aquaponic context associated with a plant growth or production measured outcome being statistically compared. A difference in a plant production or growth parameter is the most often applied comparison to determine an assumed difference within an aquaponic system context (e.g. comparison of one aquaponic system to anothercoupled v decoupled aquaponics, etc.) or via a comparison to an external, differentiated context (e.g. comparison of aquaponics to hydroponics).
The outlined Decision Matrix (Figure 1) will not be applicable to each and every experiment performed using aquaponic systems. However, in the context of using a plant measure outcome as the determinant of establishing a difference in treatments where aquaponic systems are being tested, it is expected it should account for most situations.

Study Analysis and Descriptive Statistics
It must be noted that a specific search technique was applied to identify studies suitable for this analysis (see Materials and Methods section). While the author believes a relatively high proportion of aquaponic studies using a plant growth measure as the comparative determinant (or one of the comparative determinants) were identified and included, it is acknowledged there will be studies that were not identified or included here. Ayipio  The descriptive statistics (Table 2) illustrate several interesting points associated with aquaponic system research and experimentation. The vast majority of studies (fifty-two of sixty-one studies -85 %) used a form of fully recirculating (coupled) aquaponic system and these studies applied replication correctly to the highest proportion (44 %). Only two studies used true decoupled aquaponic systems and none (0 %) of these studies applied correct replication. Of the seven studies that irrigated collected RAS water as a nutrient solution (i.e. no fish present in a direct hydraulic link to the plant component), only one (14 %) applied correct replication.
Overall, 61 % of all studies applied replication incorrectly or not at all. Hulbert (1984) points out that 48% of the ecological field studies he examined which applied inferential statistics (e.g. identification of significant differences) contained pseudo replication. Thorarensen et al. (2015) points out how little attention is given to experimental design in aquaculture fish growth experiments, especially in terms of the number of required rearing units for adequate treatment replication and Araujo (2008) argued there is a lack of appreciation of basic statistics in aquaculture experiments. Ruohonen (1998) also points out that a lack of tank replication in fish aquaculture experiments is common, leading to a lack of independent statistical outcomes or pseudoreplication. Raudonius (2017) found that almost 55% of the articles associated with crop research he examined applied experimental design and statistical analysis incorrectly and Kramer et al. (2016) found a similar result of almost 50% for crop studies. Therefore, the outcome of the current study is not unexpected, and while researchers of aquaponics apply statistical analysis to their work, it appears it is more likely than not that it is applied incorrectly.
The result of the current study for incorrect replication application (61 %) in aquaponic studies should be a major concern to the aquaponic research community as inferences are being made based on invalid experimental designs and statistical outcomes that do not support those inferences. To this end, determinant arguments are then being made based on these incorrect designs and analyses. Finally, these interpretations are being used to inform commercial applications of aquaponic technology.  (Medina, et al., 2016). Therefore, it is difficult to understand why such a high rate of incorrect replication is still being applied in the field of aquaponics?
An example of industry informing inferences is demonstrated within the field of decoupled aquaponic systems (DAS). Arguments from researchers within this field are regularly made in terms of the inferred advantage of the decoupled aquaponic approach to plant production outcomes ( However, as the outcomes of the current research demonstrates, there was only one identified study that directly compared fully recirculating with decoupled aquaponics (with fish present in the direct hydraulic link between the fish and plant components) and that one study applied no replication (Monsees et al., 2017). The applied inference was that the decoupled approach performed better than the fully recirculating approach, as based on total fruit yields, but no mean results, no standard deviations or standard errors and no comparative statistical analyses were reported (Monsees et al., 2017). Of the seven articles that applied a RAS water or RAS water variation (e.g. complimented with additional nutrients) to a plant component, five argued that the decoupled design(s) they tested were equal to, or superior to, hydroponic controls or fully recirculating aquaponic analogues. However, again, no replication was present in any of these studies and therefore, the inferred statistical differences observed were inappropriate and unreliable (Delaide et  Blanchard et al., 2020) supported by scientific data generated from correct experimental designs and appropriately applied, comparative statistical analyses, or a perception generated from theoretical arguments?

Experimental Replication Decision Matrix:
An Experimental Replication Decision Matrix was developed to try and direct future researchers towards the application of correct replication within overall experimental design. Determining the comparison that is being performed within the overall experimental design is of paramount importance and assists to provide a broad understanding of what is being tested and assists to identify the comparison context (Hulbert, 1984;Quin & Keough, 2002).
This is generally associated with the basic question: What is being compared?
Asking what is being compared in an overall experimental design context is not associated with identifying what parameters will be measured to determine any hypothesised differences (e.g. water chemistry -pH, D.O., EC, specific nutrient concentrations, etc.; plant growth or productionplant weight, plant length, leaf area, yield, etc.; fish growth -SGR, FCR, yield, etc.). It is about identifying what variable is being compared within the experimental design (e.g. aquaponic v hydroponic, fish species variations, fish density variations, fish feeding variations, fish to plant ratio variations, etc.).
Identifying the comparison context assists in determining the final experimental context which directly leads to what aspect or component of the experimental set-up requires the replication.
The important question is: Can what is being compared be compartmentalised as a sub-set of the entire culture system?
If it is impossible to compartmentalise what is being compared as a sub-set of the entire culture system (aquaponic or hydroponic; coupled or decoupled), then the experiment is occurring within an aquaponic system (or any other culture system) context and therefore, correct experimental design requires entire culture system replication. Figure 1 shows there are relatively few comparisons where replication of only a sub-set of the entire system is valid, and these appear to be almost all related to comparisons associated with a plant component context ( (2010) provides an analysis of an analogous situation that explains this difference in a RAS fish diet evaluation context where several different diets were fed to fish when replication was applied to the entire culture system (RAS) rather than the feed treatments (Arockiaraj & Applebaum, 2010). This study demonstrates that it was impossible to determine if it was the diet treatments or the individual filtration systems that caused the differences observed (Tlusty, 2010).
In most experimental situations, entire system context replication will be appropriate. This is because most aquaponic studies attempt to differentiate the aquaponic system(s) they are testing; differentiation of one aquaponic system from another (e.g. coupled v decoupled aquaponic designs) (e.g. Monsees et al., 2017) or differentiation of an aquaponic system from a different system type (e.g. ). In these cases, at least three replicates of the entire aquaponic system and any control system (e.g. hydroponic system) would be required.
Some situations will involve using nutrient solutions (no fish; solutions arising from separated RAS, nutrientcomplimented RAS solutions and controls), which does differentiate them from entire aquaponic system comparisons, but in general, entire system replication will also be appropriate in these cases because the nutrient solution is being compared (e.g. Delaide Blanchard et al., 2020). Again, in these cases, at least three replicates of each treatment system would be required (i.e. three nutrient sumps connected to three, independent plant culture devices for each treatment).One of these example studies, that compared nutrient solutions (hydroponic control vs nutrient complimented RAS water), rather than a complete aquaponic system, acknowledged that the experimental design applied (only one replicate system per treatment) did not meet the requirements of valid replication (Goddek & Vermeulen, 2018).
For most aquaponic experiments (true aquaponic or RASderived nutrient solution variations), a good default experimental design is to adopt at least three replicates per treatment.
A key point is that, avoidance of replicating entire aquaponic culture systems or units, as well as any control culture system (e.g. hydroponic system or unit), is not a good practice in an experimental design, replication or statistical analysis context for the majority of aquaponic comparative studies. Alcarraz et al., (2018) compared a standard, recirculating, deep flow (DWC or raft) hydroponic system to a recirculating (coupled), deep flow (DWC or raft) aquaponic system using rainbow trout (Oncorhynchus mykiss) juveniles (40 fish per replicate) that produced wastes used to provide nutrition to the plants. The experiment consisted of two treatments (hydroponic and aquaponic) and each treatment had three replicates. The replicates were complete system repetitions (i.e. three independent hydroponic systems and three independent aquaponic systems). The plants cultured were lettuce (Lactuca sativa L.) and each replicate contained thirty plants. The two culture systems were statistically compared to each other using ANOVA. This study was a good example of applying correct replication when comparing aquaponics to hydroponics. The culture system (hydroponic or aquaponic) was the variable being compared and therefore, replication of the entire culture system (or unit) was the valid approach. This meant that the statistical analysis applied to identify any difference between the two systems via the parameter compared (lettuce yieldgfwm -2 ) was valid and reliable.
Lennard & Ward, (2019) compared a number of lettuce varieties (Lactuca sativa L.) and herbs (dill -Anethum graveolens L., rocket -Eruca sativa, coriander -Coriandrum sativum L. and parsley -Petroselinum crispum) grown in a Nutrient Film Technique (NFT) hydroponic system and a NFT aquaponic system using Grass Carp (Ctenopharyngodon idella) to produce wastes used to provide nutrition to the plants. This study compared one semi-commercial-scale hydroponic system (1,800 plant spaces) to one semicommercial-scale aquaponic system (1,800 plant spaces) and therefore, did not apply valid replication that would allow statistical comparison. The authors recognised the study was a crop production trial without replication of the test variable (i.e. the culture systems) and therefore, did not apply any statistical analysis and only reported mean outcomes with standard errors and percentage differences. However, they did argue the calculated percentage differences in plant yields could be used as a measure of difference and argued that in the majority of cases, the aquaponic treatment outperformed the hydroponic treatment. This study was performed in a greenhouse and would be considered a crop or system demonstration trial. In a trial context it is not valid to infer differences based on the parameters measured in the absence of replication, but trends may be highlighted (Koller et al., 2016). This study was an example of correct identification of the lack of replication and therefore, any statistical analysis would not have been valid. However, arguments that identified one culture systems superiority over the other, based on the observed percentage differences in plant yields, are worth scrutiny and not scientifically reliable (Koller et al., 2016).

Aquaponic Solution v Hydroponic Solution (no fish)example solution context comparisons:
Nicoletto et al., (2018) compared three different nutrient solutions; water from an operating recirculating aquaponics system, the same aquaponic water complimented with phosphorous, potassium and a micro-nutrient mixture and a hydroponic nutrient solution control. Each treatment was replicated three times; therefore, a total of nine separate, independent culture systems or units were used. Each culture system consisted of a nutrient tank attached to four NFT channels, with a pump pumping the solution from the nutrient tank to the channels with a gravity return. The plants tested for growth parameters (plant height, yield, dry matter) were rocket (Eruca vesicaria R) and mizuna (Brassica rapa L. spp. Nipposinica M). None of the culture systems tested contained fish. This study was a good example of applying correct replication when comparing aquaponics nutrient solutions to a hydroponics nutrient solution. Even though fish were not present in any system, and even though only nutrient solutions were compared, the experiment was still comparing different culture systems (hydroponic and aquaponic) and therefore, replication of the entire system (unit) was the appropriate and valid approach. This meant that the statistical analysis applied to identify any difference between the two systems via the parameters compared was valid and reliable.
Goddek & Vermeulen, (2018) compared aquaculture (RAS) nutrient complimented water with a hydroponic nutrient solution in a similar experimental culture system set-up to Nicoletto et al., (2018) (nutrient solution tanks attached to NFT channel arrays). They used only one nutrient tank per treatment and therefore, did not apply any replication to the two treatments. They identified that replication was not present within the experimental design ("Hydrologically speaking, this approach, however, cannot be considered as a repetition."). However, despite the acknowledgement of the lack of replication, they still applied a statistical analysis (ANOVA) to the identified plant growth parameters measured (plant wet and dry weights) and used these statistical results to argue that one system (complimented RAS water) was better than the other (hydroponic nutrient solution). This study was an example of inappropriate statistics being applied in a situation with no replication (essentially, a trial) and using the identified statistical differences to infer the superiority of one approach over the other, was not a valid approach.

Three Plant Growing Media Compared in One Aquaponic Systemexample plant component context
Salam et al., (2014) compared three different plant growing media (gravel, crushed brick and a sawdust/gravel mixture) in an aquaponics system. The experimental design utilised a single fish tank attached to six separate plant grow beds (two beds for each different media), which therefore, realised two replicates per treatment tested. Because the "system" was not being compared or tested, but the plant growing media was, the use of a single fish tank feeding all the plant grow beds was appropriate, because this design removed the possibility of the fish tank or fish culture component being a variable, and concentrated any detected differences onto the media. Two replicates are considered the lowest number for valid replication (Gomez & Gomez, 1984;Gupta et al., 2015) and therefore, the replication applied in this study was theoretically acceptable. This study is an example of good experimental design and appropriate application of replication for aquaponic studies concentrating on plant-associated component variables. However, it is suggested that three replicates would be more robust.

V. CONCLUSION
It is concerning that aquaponic researchers are not applying correct replication to their experimental designs and then making inferences based on incorrectly applied statistical analyses and outcomes. It is more concerning that these studies are being published in peer reviewed journals, which are supposed to be providing a process to expertly review and ensure the validity and appropriateness of the experimental designs and statistics applied of the studies they publish. This suggests, that a lack of understanding of correct experimental design, correct application of replication and correct application of appropriate statistical analysis is far too common within the cohort of journals publishing aquaponic articles and among the associated reviewers assigned by these journals; a situation that is seen at similar levels within the associated disciplines of aquaculture (Hulbert, 1984;Ruohonen, 1998;Araujo, 2008;Thorarensen et al., 2015) and crop science (Kramer et al., 2016;Raudonius, 2017). An experimental replication decision matrix was developed, and several example articles discussed, in an attempt to assist future aquaponic researchers to determine the replication requirements of the aquaponic studies they design.