Full transcript
0:08Hello, this is a great pleasure of uh
0:11having this presentation on responsible
0:14research and scientific integrity for
0:16the uh EU
0:20for the EU 2030 uh meeting.
0:24And uh let me begin by introducing
0:29the uh uh our own group the synopsis
0:33meta research group
0:35which uh is unique in the sense that it
0:40applies research research methodologies
0:43to the study of uh uh practices of
0:47university practices themselves. In some
0:49sense, it incorporates
0:52scientific knowledge to study research
0:54policy and uh optimize the the practices
0:58of uh uh
1:02universities themselves. Um
1:24Apologies for the apologies for the
1:27stop. I have a little bit of a sound
1:28issue. Can you please confirm that you
1:30can hear my presentation clearly and
1:32then I will uh I will
1:36>> Yes, we can hear you clearly and we can
1:37see the presentation as well.
1:40>> Great. So the synopsis group is uh uh
1:46consisting of several uh tasks. So it
1:51serves as a meta research center at the
1:53University of Cyprus. Metal research is
1:55an emerging uh research field that
1:58involves uh different disciplines and
2:01different uh research centers and
2:04interdisciplinary
2:05uh working groups in different countries
2:08again with a uh purpose of studying the
2:11faculty of science itself.
2:13Uh one of our main uh interest is to
2:17bring together other scholars at the
2:19University of Cyprus interested in
2:22research policy interested in
2:23understanding the practices of academia
2:26uh and uh basically create a forum and a
2:31group that allows discussions
2:34discussions to take place and research
2:36interdisiplinary research to be
2:38conducted but also the university policy
2:41itself to be informed.
2:44And uh we are also trying to uh generate
2:49meaningful policy advice. So this kind
2:52of uh uh coalitions and this kind of
2:56initiatives like uh EU they have this uh
3:00uh special features of uh outline the
3:04importance of uh young research
3:07universities in Europe and how they can
3:10be special. they can be pioneers in uh
3:13uh new approaches to the research
3:16policy.
3:19So uh once more welcome to our uh uh
3:25to our guests and especially the the the
3:28junior people interested in uh research
3:30practices and integrity. So this is part
3:33of course of the uh youth training
3:36series on responsible uh research
3:38practices and this is the uh domain that
3:42we're going to be uh touching upon.
3:45Uh importantly we will speak about what
3:48might go wrong in uh science and which
3:53are the approaches that have been
3:54discussed in order to improve things.
3:57Basically when we all start as
4:00researchers, we start a little bit in
4:03the idealistic view that research is the
4:07search uh that science is the search of
4:09truth that it is an impassionate process
4:13of discovery. And then the more we uh
4:17work and we experience uh actual
4:20research, we realize that science is of
4:22course as we should have known in the in
4:25the first place a a human institution
4:28uh with its uh advantages and
4:30disadvantages. And therefore uh we
4:34should be careful and treated in the
4:36same way as as other institutions with
4:39uh weaknesses and uh uh potential
4:43policies to uh amilarate this.
4:48So today we will focus on the
4:51problematic uh of uh
4:56credibility crisis in research. the
4:58antidote to this which are good
5:00practices involving transparency and uh
5:04leading to better reproducibility and
5:07also we will also focus on an
5:09interactive approach. I hope my uh sound
5:13does not look very problematic there
5:17where we will illustrate uh the ideas
5:20the particular ideas about uh
5:23problematic versus good practices with a
5:26real example
5:29and again the idea is to be a sort of
5:32self-contained
5:33uh material and uh tools that can be
5:37applied uh practically
5:40So in particular in terms of the
5:43learning objectives
5:45after the end of this uh meeting you are
5:50expected to be able to understand the
5:53replication crisis in science and its
5:55implications to learn about
5:59specific practical open science tool.
6:03For instance, the new developments in
6:05pre-registration and the open code uh
6:08sharing often in real time uh should be
6:12able to apply uh responsible metrics.
6:17responsible metrics are closely related
6:19to the problem of uh the quality of
6:21science, reproducibility, prices and so
6:24on and so forth because essentially
6:27uh the incentive of scientists how they
6:29are rewarded affects their behavior
6:32especially from an economics
6:33perspective. Incentives are the most
6:36important uh element in uh an economic
6:39behavior. So how metrics are used to
6:43assess uh the quality of research is of
6:45fundamental importance for good science
6:48and that's that is where the concept of
6:50responsible metrics goes in uh for
6:54example as as applied by uh the Kara
6:58initiative
7:00and related to that going a little bit
7:04uh deeper into the concept of
7:07responsible research evaluation we will
7:10also mention mention the uh what these
7:13are and how they can be used to showcase
7:17diverse contributions. These measure
7:19such narrative are important especially
7:22today that modern tools allow us to
7:25reduce the cost of producing a high
7:28quality narratives of our uh
7:32contributions and essentially explaining
7:34the research journey of the scientists
7:36rather than bullet points with
7:38individual obligations.
7:41And finally,
7:44you should be able to recognize and
7:45address integrity uh challenges with
7:48research.
7:50And of course, uh the issue of integrity
7:54uh is closely related to almost all of
7:56the aspects that we have uh discussed so
7:59far and the ultimate quality of uh
8:01research uh being produced by science.
8:06So when I was uh involved in this topic
8:08of the uh of meta science, my first uh
8:14contact if you wish uh was uh an article
8:18at the New Yorker which had this uh this
8:21particular image which talks about the
8:25truth wearing off. So about 15 years ago
8:29to 20 years ago there was a
8:33initiation an initiation of a discussion
8:36about the potential replication crisis
8:39in the sense that uh it seems that
8:42subsequent studies that started from
8:44some often famous fightings did not seem
8:48to find similar results. In other words,
8:51uh the initial results did not seem to
8:54be uh replicable.
8:58And uh this pattern in the following
9:00years since the 2010s has uh exhibited
9:05itself across different disciplines. And
9:08uh the most systematic way to show the
9:12problem was large replication
9:14initiatives. initiatives that brought
9:16together
9:18uh collaborators in different
9:20universities, often a very large number,
9:22that conducted a great number of
9:24replications
9:25uh in a coordinated way. That was the
9:27best way to actually uh bring things
9:30together and show clearly the the
9:33picture which was that in uh important
9:36social science disciplines such as
9:38psychology
9:40uh about one/ird of studies are
9:42replicated.
9:43So which means that for uh starting
9:46researchers uh from Eupet let's say in
9:49the domain of psychology or social
9:51science you should be um contemplating
9:55on the underlying reasons why uh
9:58although as scientists we're supposed to
10:01to produce uh robust and uh uh
10:05generalizable knowledge we can often
10:08find ourselves in this uh difficulties.
10:10So we're talking about the work of
10:12established
10:13uh very distinguished uh sometimes
10:16researchers
10:18uh who were conducted the initial
10:19studies but nevertheless uh the large
10:23initiatives in replication showed that
10:25these results were not uh replicable
10:30and uh along more or less along these
10:34lines and at the same time similar
10:36initiatives happened in in economics.
10:39one led by Colin Came and his team which
10:43shows that economics fared best. There
10:45are many reasons why one would consider
10:49one discipline to be more ethical than
10:51than the other. We didn't have time to
10:53go in into that depth but in
10:56experimental economics the uh
10:58reproducibility
11:00rate was uh better.
11:03And then of course uh the poster child
11:07of the discussion about the
11:09reproducibility crisis was biome
11:11medicine. It started all this. Um so
11:15already since 2012 there were some uh uh
11:20famous uh reproducibility initiatives to
11:22replicate very important findings in
11:25cancer research with very low
11:27reproducibility rates. And uh often
11:31times it was pharmaceutical companies
11:33that conducted this uh replication that
11:36showed the uh often disappointing
11:39reproducibility rates for uh published
11:42preclinical studies.
11:46So now that we set the stage and showed
11:50that uh
11:53things do not go always in a rosy and
11:56expected way in science but there can be
11:59problems. We can go a little bit uh more
12:02deeper and think about the underlying
12:04reasons for these problems. So uh for
12:08starting researchers for junior
12:10researchers it is important to think and
12:12contemplate about the driving factors of
12:15nonreproducible research. Uh the general
12:18discussion has taken place across
12:20several uh general categories including
12:23theological flexibility.
12:26uh this roughly means that you're
12:28allowed to make conclusions on the basis
12:31of analysis often statistical analysis
12:34that takes place a lot of different
12:36forms. So the more experienced you one
12:39becomes with statistical analysis the
12:40more we realize that there are very many
12:42different ways to analyze a given data
12:45set and often times this can lead to
12:48different conclusions. This is the
12:50concept of methodological flexibility
12:52and it is problematic because often
12:54times as researchers we know if you wish
12:57what is the desirable outcome in terms
13:01of propagation prospects which of course
13:03is is problematic.
13:05Uh similarly the lack of transparency of
13:09uh the methods is if you wish
13:11interactive methodological flexibility
13:14uh exacerbating things and then uh this
13:19all these play a role um together with
13:24statistical noise which is if you wish a
13:26natural reason why results would fail to
13:29be uh replicable.
13:32And uh ultimately
13:35in some uh rare cases there might be
13:38even a research misbehavior in terms of
13:41uh uh conscious problematic reporting of
13:45the evidence because of uh incentives to
13:48find the results in a certain uh in a
13:51particular uh direction.
13:53And uh also milder forms of research
13:57misbehavior relate to the lack of
13:59transparency.
14:00uh those are called are referred to as a
14:04question of research practices whereas
14:06whereas research behavior is more
14:08related to conscious uh fraud.
14:12Okay. At the at the same time at the
14:16institutional level the macro level
14:18there are factors such as uh publication
14:20bias. These are not directly related to
14:23the practices of individual researchers
14:26but uh about systemic incentives in uh
14:31the publication process. uh it's not
14:34just individual researchers who are
14:35interested in getting interesting
14:36results but of course uh it is the
14:39publishers themselves and the editors
14:41themselves what are interested to uh
14:46declare interesting discoveries in terms
14:48of their readability their scientific
14:50impact and as a result publication bias
14:53refers to phenomenon whereby what
14:56actually ends up being published is only
14:58a biased sample of the actual uh
15:02research
15:03biased in the way that significant
15:06results are over represented. In other
15:07words, findings that declare a new
15:09discovery, some association between
15:12variables important for scientific
15:15knowledge. These kind of discoveries are
15:17often uh heavily uh over represented in
15:22the literature. But the failure to uh
15:26confirm an association or a discovery
15:29that is much uh less interesting uh if
15:33you wish but it's not less important for
15:35science and this bias creates important
15:38problems in seeing the totality of what
15:41we should be knowing about uh uh
15:44different uh variables and relationships
15:47in nature.
15:50Now be hacking is a sort of more
15:52concrete notion related to the notion of
15:55methodological flexibility.
15:58Uh the more we gain experience with
16:00statistics, the more we realize that in
16:03practically every scientific uh
16:06empirical article there is a large
16:08number of statistical tests. There's a
16:11large number of interesting variables.
16:13So usually there is like a general uh
16:17interesting uh relationship that we want
16:20to examine but how you operationalize
16:23showing that this relationship holds
16:25often takes place in a complex way using
16:28different variables different dependent
16:30variables different independent
16:31variables even. So there are very many
16:34ways that you can show evidence
16:36corroborating the uh fundamental
16:40relationship. So often times what can
16:44take place is that there is a choice of
16:49those statistical tests that seem to
16:51work that seems to show uh a a
16:54relationship which is interesting and uh
16:59uh most likely publishable and uh hiding
17:03the potentially great number of
17:06statistical tests that was not
17:07supporting this uh this relationship.
17:10This is the concept of B hacking and of
17:12course it's very closely related to the
17:14notion of uh transparency
17:17and the lack of transparency and
17:18methological flexibility because
17:20obviously if there is enough
17:22transparency then you cannot selectively
17:25choose which uh statistical tests to uh
17:29to present in the final
17:32and then hacking is is related in the
17:35sense of uh
17:38uh the initial study being uh
17:42not sufficiently driven by theory. So it
17:44is important as scientists that we uh
17:48advance theory in our discoveries.
17:50Essentially theory is the tool by which
17:52we learn about uh truths in nature.
17:57Science essentially is theory mechanisms
18:01uh of uh our understanding. So if we
18:04don't start with strong enough theory,
18:07we might result describe the result is
18:10harding or hypothesize after the results
18:12are known. Then you don't have two
18:14hypothesis after observing a large
18:17number of statistical tests. You might
18:19decide that
18:22it seems that there is some pattern in
18:23the data. So you uh post talk theorize
18:28what might be causing what and you might
18:31even present this as a hypothesis that
18:33we had in the first place which is of
18:35course is disingenuous.
18:39So again uh researched degrees of
18:42freedom is related to most of these
18:44issues at the individual level
18:46and lack of transparency.
18:50Of course, this is a fundamental problem
18:53and this is a why meta research, meta
18:56science as a new domain focuses so much
18:59on this
19:02and uh those important consequences are
19:06of course the wasted resources for
19:09society of investing and building on
19:12false findings for instance the
19:14development of non epicious drugs.
19:19uh it has been estimated that this this
19:21amount is uh extremely high at the rate
19:25of uh 28 billion per year according to
19:28estimation.
19:30Uh there's also more fundamental long
19:33run uh erosion of the public trust in
19:36science. Uh, of course we're living in
19:39an era where unfortunately we see a lot
19:42of uh uh aspects and a lot of uh
19:47appearances of this problematic uh
19:49phenomenon. The erosion of public trust
19:51in science.
19:53Uh often populist movements uh utilize
19:56uh this for their own interests.
20:00And of course these things are
20:02translated of
20:04uh inevitably in delayed uh process in
20:09addressing societal problems. So issues
20:12that could have been resolved faster if
20:15knowledge were was able to accumulate uh
20:18better without these biases now uh are
20:22not addressed as they as they should.
20:26And then you have the implication for
20:29career impacts. Often times we observe
20:33uh scandals and research uh non
20:36reproducible findings scandals about
20:39behavior and these reverberate on the
20:43scientific careers of juniors and senior
20:45researchers. Often times there are as we
20:48shall go into this later
20:51there are cradles and uh there are uh uh
20:55difficult decisions that need to be made
20:57within a scientific team and uh
21:02this is very important for career
21:04developments and uh
21:07in terms of ethical choices. Uh actually
21:11within synopsis we have uh economic
21:14theory models that try to examine these
21:17different incentives of junior and
21:20senior researchers and how uh the
21:23responsibility for research behavior can
21:25be optimally allocated in order to uh
21:30optimally solve this problem of uh
21:33genuine revelation.
21:35uh in cases of uh recent misbehavior by
21:39uh other parts of of a given group.
21:44So now in this part we will speak about
21:48uh more in depth about meta research
21:51study research itself. So the founding
21:55uh documents the hub of the domain of
21:58meta research was the very famous uh uh
22:02publication by John Eidis
22:05in 2005 with a very provocative title
22:09why most published recent findings are
22:12all.
22:13So this came rather suddenly into a
22:17scientific uh reality and society that
22:22was not alert to such an important
22:24problem of uh lack of reproducibility
22:27and John wanted to make this strong
22:30statement. Actually uh Ian and his team
22:34along with other researchers for several
22:37years before 2005 were conducting and uh
22:41advancing the domain of meta analysis to
22:44show how knowledge accumulates and they
22:47were they had gathered quite substantial
22:50evidence about the problems there but uh
22:54society and science at large not
22:56acknowledged the issue. So therefore
22:59this was sort of a manifesto rather than
23:01a pure scientific article in 2005 and it
23:04initiated
23:06uh the domain of meta research. But what
23:09exactly uh is meta research? So this is
23:13John and uh he along with others has
23:17defined meta research is the stud study
23:20of science itself.
23:22To be fair, the study of science has
23:25been uh studied for more than seven
23:28eight decades in mainstream research in
23:32the domains of philosophy of science, uh
23:34sociology of science and uh so on and so
23:37forth.
23:38Uh however, what is sort of new in this
23:41domain of meta research is the highly
23:43interdisciplinary aspect. So now
23:46researchers from different disciplines
23:47come together to understand the workings
23:49of science and it is also very
23:52quantitative is closely related to the
23:55statistical area of meta analysis
23:58uh which which are very strong ways in
24:01which it diverges from traditional
24:03philosophy and
24:06sociology of science but also
24:10uh it is uh determined as a solution
24:15to a important topical phenomenon which
24:18is the credibility crisis in science. So
24:20these are characteristics that uh define
24:23it uh and differentiated from
24:25traditional approaches.
24:29So what does meta researcher study? It
24:33studies the different uh science
24:35methodologies whether they are
24:36appropriate whether they are robust
24:39uh whether they can be improved. Okay.
24:43Uh similarly, it's not just the method
24:45that can cause biases and problems in
24:47science. It's also the reporting of the
24:49results.
24:51Uh we we referred to this previously. Uh
24:54selective reporting, dehacking,
24:57uh empirical
24:59uh
25:01examination of the extent of this
25:03phenomena. Uh potential remedies, all of
25:07those are within the domain of uh meta
25:09research.
25:11uh measures of reproducibility as we saw
25:14before uh
25:18solutions to the problem of
25:19reproducibility
25:21uh interventions
25:22policy interventions to uh improve them
25:26and of course research evaluation and
25:29assessment. How can we optimally assign
25:33uh rewards in uh in research such that
25:37they uh
25:40they improve on the problems of bias,
25:44honorable disability and so on and so
25:45forth. Align them better with the
25:48underlying objective of uh scientific uh
25:51discovery of the truth in in nature.
25:56Okay. And again as I hinted to before in
26:00those uh all those domains empirical
26:02methods play a very crucial role. So
26:05these are not just uh uh generic uh
26:11verbal uh discussions about how uh
26:15science works but a much more focused
26:18empirical approach.
26:22Okay. And as we we mentioned before,
26:25we're talking about
26:27uh measuring the degree that problems
26:29exist and the uh possible solutions and
26:33interventions.
26:35And uh importantly for junior
26:37researchers, it would be important to
26:39know that meta research is a growing
26:41field that attracts increasingly
26:44uh the support of funders and
26:46international organizations obviously
26:49because of the importance of the waste
26:52in uh uh public funds that come from the
26:55problem of reproducibility. So as a
26:58result uh there's a great state for
27:00society to improve things and getting it
27:03right. That's why meta research is uh
27:06strongly supported in
27:09Europe and beyond and that's why we see
27:12and we we hope and synopsis that we have
27:15held there have held there that
27:17initiatives that incorporate open
27:19science aspects that is UK itself is
27:23starting to incorporate research
27:25research within its uh domains the same
27:28with uh with kara
27:35So there's a large literature in meta
27:38research that provides specific examples
27:41of what we said before uh problems in
27:44science uh problematic methods
27:48problematic evaluation
27:50uh criteria and so on and so forth.
27:54So there's a large number of studies
27:57actually extremely large uh within the
28:00meta analysis literature that shows the
28:03problematic effect of uh small sample
28:06sizes especially combined with
28:09publication bias. You reach a situation
28:11where you have a proliferation of
28:13studies with small samples declaring
28:15large treatment effects. large
28:19uh associations between variables in
28:22science that are often times very weakly
28:26reproducible.
28:29Um then the the recent solutions that
28:33have been suggested for uh these
28:35problems include pre-registration.
28:38Pre-registration is important. We will
28:40get into more uh depth but uh roughly
28:44speaking
28:47If the problem is the researcher degrees
28:50of freedom, how a researcher can conduct
28:52a large number of statistical tests and
28:55then report only uh a fraction of these
28:58tests that seem to be favoring uh their
29:02own uh objectives in terms of what is
29:04interesting and publishable. Then what
29:07we should be doing is tying our hands
29:09essentially saying beforehand what kind
29:12of u test we will be reporting exactly
29:16which kind of hypothesis we have even in
29:20case of experimental research how we're
29:22going to be generating new data uh and
29:26how exactly our uh studies will be
29:28conducted with what sample size and so
29:30on and so forth. So registration is uh
29:34one of the most important and popular
29:37suggested remedies to the problem of
29:40research reproducibility
29:42and it has been found uh relatively
29:45recently by uh strong research teams
29:49among them the team of Abel Bier who has
29:52an institute of for replication at the
29:55University of Otawa who plays a large
29:57role in social sciences that indeed
30:00registration plays an important role
30:02role in reducing uh false positives.
30:06False positives are findings that are
30:08declared as true in the literature.
30:12This for instance this uh uh drug tends
30:15to uh
30:18reduce certain forms of cancer. This is
30:20sort of a uh scientific declaration.
30:25This is a false finding. false finding
30:27false positive excuse me to the degree
30:30to which it has been published in the
30:32literature that there is this
30:33association this uh efficacy of the drug
30:37which is actually false okay so
30:40registration tends to reduce the rate of
30:43false positives misleading findings in
30:46the literature
30:48uh in addition there is increasing
30:51evidence about how the
30:54additional suggestions about open data
30:57uh improve the the degree to which u
31:03scientific studies uh adhere to high
31:07standards of uh analysis, transparency
31:10and methodology.
31:13So the the new uh policies of open data
31:19where each uh researcher should be
31:23particularly revealing their data to the
31:26degree possible to the uh society and to
31:29the scientific community to be verified
31:32plays a positive role and has there have
31:34been some recent studies there.
31:38Okay. And in addition, this is an even
31:40older literature
31:42that shows how problematic is the degree
31:46to which publication bias exists, how
31:48widespread it is across disciplines and
31:51how important are the limitations of uh
31:54peer review.
31:57Uh this is important because peer review
31:59is the fundamental cornerstone of uh
32:02modern science. Uh what is considered
32:06significant enough? what is considered
32:09of sufficient quality in order to be
32:12added to the corpus of scientific
32:14knowledge.
32:17This essentially has been
32:21that decided using the institution of
32:24peer review. Uh similar researchers
32:27decide on the
32:30uh on what should be uh revealed
32:35uh
32:37on what should be published, what should
32:39be uh added to the corpus of knowledge.
32:42And uh this is the way this is the
32:45safeguard of uh quality and uh uh for
32:49science.
32:54And in addition, this is also a current
32:56work we have conducted in synopsis and
32:58with in other research groups
33:03uh examining how incentive structures uh
33:06matter for science rewarding quantity
33:09over quality. For instance, this is an
33:11important uh underlying
33:15uh problem behind the initiative and uh
33:20how to improve this uh current
33:22incentives is a is a key challenge. But
33:25first meta research comes forward and
33:29uh shows how this uh incentive
33:32structures reward
33:34quantity over quality.
33:47So at this stage,
33:50let me
33:52let me ask you a little bit uh whether
33:57you have been able to uh to follow the
33:59presentation so far and whether there
34:01are any uh general questions to be
34:04addressed.
34:12Are there any questions so far?
34:21>> No.
34:23>> All right.
34:25>> Very good.
34:26>> Can you please just now that we we have
34:29a small interruption, can you please
34:30press this hide hide in your screen
34:34because we can't see
34:37the full now we have the full
34:43>> thank you. So uh
34:49as I hinted uh before a lot uh after the
34:54discipline of meta research has matured
34:58one of the most important uh uh aspects
35:01was to uh find solutions to these
35:04reproducibility problems. Make sure that
35:07researchers who start their career now
35:11start with the right foot on how to
35:14follow uh best practices, how to conduct
35:18their obesity research, how they don't
35:20fall into the trap of uh
35:24excessive uh flexibility, lack of
35:26transparency
35:28uh no and the lack of open research
35:32which uh to a large degree we have been
35:34at hold for the problems that were uh
35:37substantiated before.
35:40So uh for instance here you can see two
35:44key platforms
35:46where uh in a systematic way and with
35:50systematic guidance modern researchers
35:53can uh upload not only their data but
35:57also their research protocols,
35:59methodologies,
36:01analysis and code uh publicly available.
36:04So this is a standard practice that uh
36:08many researchers who are inspired by uh
36:13the meta research movement are
36:14following. So in other words, whenever
36:16there is a new article
36:20uh either in the form of final
36:22publication or a working paper, it is
36:24always accommodated by the link at the
36:27open science foundation or
36:30at GitHub where all this information is
36:33uh is available
36:36and I have to emphasize that with the
36:39development of AI these uh sort of
36:42documentations which are traditionally
36:45considered burdensome and uh taking time
36:48and reducing the uh the potential
36:52productivity of researchers those will
36:53be streamlined and much faster to do so.
36:56So that will be something that
36:57technology will do for open science and
37:01uh I'm very optimistic about that uh
37:03making all this uh a standard practice
37:06and relatively low cost of time for
37:10scientists.
37:12So let's go into maybe uh the most
37:16popular approach in experimental science
37:20uh about how to deal with the problem of
37:24uh uh non-reproducible research bias
37:28uh flexibility methodology as I hinted
37:32before uh this uh
37:36this method is pre-registration and uh
37:38accompanied with a pre-analysis
37:41Pre-registration
37:43essentially says that before conducting
37:46an experiment you publicly reveal that
37:49you will do so you publicly reveal your
37:52hypothesis what you want to examine uh
37:55by this by this experiment the sampling
37:57process who are going to be the
38:00participants how they will be chosen
38:03uh the sample sizes so all the details
38:07about the uh data generation process
38:10This part is pre-registration and the
38:13pre-analysis plan concerns the
38:16specification of the future statistical
38:19analysis data analysis that will be uh
38:22conducted.
38:24So currently as we as we know so far
38:28from meta research uh uh evidence
38:33registration together with a
38:34pre-analysis plan is feasible in uh
38:38experimental research. Again there is an
38:40active discussion and we are also
38:42participating in that to what degree
38:46uh reanalysis plans tying our hands on
38:49the uh statistical analysis we conduct
38:53whether it can be used to non uh
38:57experimental empirical analysis. But
39:01generally this is this is difficult
39:02because as long as the data have been
39:05out there it is relatively hard to
39:08verify almost impossible to verify that
39:10the analysis have not been conducted
39:12already and then selectively
39:16uh revealed in a pre-analysis plan. It's
39:19not truly a pre-analysis plan. So there
39:21are issues of trust there. As long as
39:23the data have not been generated yet,
39:25this can be quite uh a substantial
39:28improvement.
39:30Okay.
39:33So one closely related concept that uh
39:38many of uh our junior researchers here
39:40should be very conscious about is the
39:43concept of confirmatory from exploratory
39:45research. So in other words when we
39:49conduct a new empirical study we should
39:51always start with some theory some
39:53strong theoretical hypothesis. This is
39:56called confirmatory research. As long as
39:59we do that, we have strong hypothesis,
40:01then we can use the usual interpretation
40:05of statistical tests and statistical
40:06evidence as testing a hypothesis
40:09providing us with a strong support as
40:12long as our significance level is
40:14particularly high. In other words,
40:16unless this association that we have
40:19hypothesized existed, it would be very
40:22unlikely
40:26that we would have been able to observe
40:28this kind of uh
40:31data patterns.
40:34Okay. But in order for this whole
40:36statistical process to work out should
40:38be based on clear hypothesis.
40:42Okay. If you don't have a hypothesis but
40:44you have a large number of regressions
40:46with all sorts of uh
40:49statistical tests being conducted on uh
40:52uh
40:54on different uh on different domains and
40:58uh you you cherrypick what kind of tests
41:02you you basically report. This is a pure
41:04violation of the the principles of of
41:07statistics because beforehand you should
41:10have hypothesized clearly on the
41:13direction of effects.
41:15So in summary
41:18confirmatory research starts from
41:20rigorous theory rigorous uh hypothesis
41:25supported by by pre-registration
41:28especially.
41:30So when you have this kind of findings
41:32in the literature where you have uh
41:35previously clearly made your hypothesis
41:39uh salient and transparent and the
41:42results uh supported this this is
41:44confirmatory research. It's much
41:47stronger than exploratory research which
41:50is the second category where you don't
41:52have uh clear evidence. I'm sorry where
41:56you don't have strong hypothesis.
41:59So as I told you before you conduct very
42:02many uh statistical tests in in a given
42:06analysis some of them show you
42:08interesting patterns but these were not
42:10hypothesized. We don't really know why
42:12they are happening but nevertheless they
42:14might be interesting in order to reveal
42:16in a scientific article maybe for future
42:19exploration. The important thing is as
42:21young researchers uh what should be
42:24conscious is to be clear, transparent
42:26and uh honest regarding which of the
42:29results were confirmatory and which were
42:32exploratory. This is this is the key and
42:34current in research practices this plays
42:37an important role
42:40and of course I think it should be clear
42:42by now that these strategies of
42:43prehacking and targeting that we said
42:45before cherry picking which results to
42:48reveal uh making a
42:52theorization after the results are known
42:55those are going to be ruled out as long
42:57as there is pre-registration in
43:06in practice for pre-registration
43:08including the open science uh uh
43:10framework as predicted uh the an
43:14American economic association randomized
43:16control trials registry and so on and so
43:18forth. So there are several options and
43:21importantly for starting researchers
43:24there's a guided process in these
43:26platforms about how to conduct uh
43:29pre-registration and pre-analysis plans.
43:31So to make sure that you can proceed and
43:34you have a complete uh information about
43:39your studies
43:44as I mentioned before
43:46uh in terms of pre-registration what is
43:48clear is the initial uh research
43:52question how many subjects are going to
43:55be chosen and how uh inclusion exclusion
43:59criteria and going uh
44:04and going further into a pre-analysis
44:07plan. Uh
44:11we we will be adding information about
44:13the actual analysis will be conducted,
44:15which are the variables that will be
44:17measured, how they will be measured.
44:20uh and of course the the specifications
44:22of the uh statistical analysis which
44:25exact tests are going to be used what
44:28will happen in the cases of uh outliers
44:31and so on and so forth. So this is the
44:34new tendency very strong and for several
44:37domains of research uh it's actually
44:41increasingly enforcable which means that
44:44uh for important uh uh aspects of the
44:49reputation of the researcher the ability
44:51to publish and so on and so forth
44:53registration plays an important role. So
44:55it is important for junior researchers
44:57to start utilizing as soon as possible
45:00preferably for the totality of uh of
45:03their careers and of course under the
45:05guidance of more uh senior researchers.
45:10Another uh popular approach is the uh
45:14the use of open methods and code. Uh so
45:17this is part of the more general
45:21uh movement for uh for open science.
45:27So it has been used for a uh for a long
45:30time for uh
45:32um
45:37uh for a long time. It has been
45:39discussed in the uh in the open science
45:43uh movement from the perspective of uh
45:47h
45:49from the ethical perspective if you wish
45:51that openness is good. So it is
45:54important for all the evidence to be out
45:55there. There's no reason to have uh
45:58science uh results methodologies hidden
46:02not open to the public. Meta research
46:06after it has uh come to the to the front
46:10have shown also the consequences of not
46:13having open methods in terms of the
46:15credibility crisis. So now the two
46:17things have come together. The general
46:20philosophical principles, ethical
46:22principles of open science for having
46:24more openness and the more practical
46:27advantages of open uh science in terms
46:30of uh increasing their reproducibility
46:32of results.
46:39So these are sort of more technical uh
46:44details and guidance on on open methods
46:48uh using modern tools such as uh Jupyter
46:52notebooks. Uh it is possible to uh
46:57document this the processes of the uh of
47:02the coding uh methods much more detail
47:05and often times even in in real time to
47:08make everything transparent, make
47:10everything reproducible to make sure
47:13that we know exactly how to produce the
47:15same results. And it is important to
47:18emphasize that uh
47:21oftent times uh this seems like an easy
47:25process to reproduce using the same code
47:27but it can be very difficult when you do
47:30it in practice. from per personal
47:33experience. Uh
47:36these uh reproducibility
47:39checkins
47:40uh checks, excuse me, when you have uh
47:45been uh successfully
47:47uh successful in publishing your work in
47:50uh in top journals uh are increasingly
47:55adopted. In other words, the journals
47:57come back and say, "We are interested in
48:00your study. This is good. We will uh
48:02publish it. But we kind of need to uh
48:09uh conduct a reproducibility check
48:11beforehand.
48:12Okay. And then when often times this
48:15process takes months and when this
48:18happens often differences from the
48:20published results to the uh to the uh
48:26reproducibility efforts uh to the
48:29replication efforts excuse me come to
48:31the front. In other words, we reproduce
48:35we we produce some initial results. We
48:38publish them in a paper. We describe
48:40them and sometimes when you want to
48:43reproduce them just prior to publication
48:45there are differences sometimes small
48:47sometimes large.
48:50All those issues would be avoided if in
48:53the beginning of a research progress uh
48:56process of uh methods open methods and
49:00code are implemented.
49:04Okay. If from the beginning the
49:06protocols are shared, the analysis is
49:08publicly available the workflows are
49:10reproducible.
49:12These things that are actually
49:15now imposed uh exposed
49:18would be available could be resolved uh
49:22exactly. Okay. So again once more for
49:25junior researchers it's important to
49:28start now by being very conscious about
49:32the importance of sharing methods and
49:34codes publicly in order to get more
49:36feedback in order to get more
49:39recognition about the open practices in
49:41order to be prepared for this internal
49:43application process that I just
49:45described and of course because once
49:48more uh the cost of these adjustments uh
49:52strongly decrease ries with the
49:54artificial intelligence methods and I'm
49:58optimistic that these things will be uh
50:00very still soon mainstream with a
50:03fraction of the cost that they use uh to
50:05have
50:08okay and of course often times small
50:11differences in the replication process
50:14come from uh different software versions
50:17and dependencies and all of those things
50:19will be strongly smooth smoothed out by
50:22having the methods are probably publicly
50:24available.
50:26And of course, this will be very
50:28important in the uh process of uh
50:31catching errors both for the feedback
50:33but most importantly for the process of
50:36cleaning our code to make it uh
50:39presentable, understandable to uh to
50:43somebody else that will allow us to
50:45catch most of the uh of the problems.
50:50I will have to apologize at this moment.
50:52I would really need to make a a small
50:55water break. Uh I will be back in in one
50:58minute. Once more, please kindly accept
51:00my apologies.
53:47Thank you so much for your patience.
53:49Apologies for the uh for the break. Uh
53:54can I ask you if there are any issues or
53:55any questions that might memorize?
54:03Everything good?
54:04>> All good, Professor Man. Thank you.
54:07Okay.
54:10So uh
54:12we'll get into the particular uh problem
54:16of open data which means uh um
54:21statistical uh uh quantitative
54:24information about uh uh phenomena in
54:28nature codified into
54:31uh into data when can they be publicly
54:35revealed? uh what would be the
54:37advantages and disadvantages of doing
54:39so. From the perspective of open science
54:42uh it is clear that uh this would be a
54:46beneficial process to have uh data as
54:49publicly available as possible. At the
54:52same time it should be acknowledged that
54:54uh uh
54:58there are some limitations often times
55:00legal implications in the uh revelation
55:04of data depending on how Of course they
55:06were uh produced because obviously data
55:09can be uh used for uh
55:13having financial and otherwise
55:16advantages
55:18uh by exclusive exploitation of the data
55:20and therefore the data is a sort of
55:23unique aspect of the uh of open science
55:27that needs to be having special
55:29consideration.
55:32So some of the ethical considerations
55:34that should be uh minded when you think
55:37about open data is the issue of
55:39identification. Sometimes you have
55:42participants in experiment for instance
55:44and uh they should not of course be
55:46identified. There should not be uh
55:49revelation of uh private information
55:53regarding their behavior uh in economic
55:56experiment for instance. Uh
56:00so
56:01it's important to share this data but in
56:04a deidentified way.
56:07Of course an important uh advantage an
56:11important benefit of open data is sort
56:13of uh clear by our discussion so far
56:16allow for verification the reanalysis
56:20and the summary of the overall evidence
56:22by means of meta analysis by pulling uh
56:25data from different studies together and
56:28from the individual perspective of the
56:30researcher if we make our data publicly
56:33available it allows for further analysis
56:37increasing also our citation rates and
56:39and impact.
56:42But there this is a domain where the new
56:45generations of researchers should be
56:47more conscious and know more about the
56:50uh legal issues of it. Okay. Issues of
56:54privacy, consent, sensitive information.
56:57uh how from the technical point of view
57:00to use the appropriate repositories how
57:02to make the appropriate rights uh for uh
57:07uh for the use of the data. So these are
57:09aspects that uh it is clear that the new
57:14generations of researchers should be
57:16more uh prepared to take into account
57:19and I think that the more time passes
57:21the education the scientific education
57:24of junior researchers should incorporate
57:26these things making them more mature to
57:29the legal issues uh behind it. So for
57:33instance uh when we uh have a uh grant
57:37application when we receive funding for
57:40our uh work we are often asked to have a
57:44data management plan which addresses all
57:47these considerations
57:48making things as open as possible but at
57:51the same time respecting the uh legal
57:55and ethical aspects and uh showing uh
58:00from a technical point of view how it is
58:02physical that you achieve those
58:03objectives. So it is important that in
58:07uh the modern uh uh development of
58:11junior researchers these things are not
58:14learned as our generation uh empirically
58:18when we had a only when we had a uh
58:22funding application process but a modern
58:26scientific education makes uh people
58:29more conscious of this of these concerns
58:31earlier.
58:36So to all the issues that we mentioned
58:38before uh an important role is played
58:42from individual incentives.
58:46Uh so why do uh researchers
58:51conduct these problematic practices? Why
58:55are they not always?
58:58Why don't they al always uh make data
59:01openly available when necessary? Why
59:05don't they always make code available?
59:08Why do they engage in practices such as
59:10P hack? All of those things can be
59:12traced back to the incentive structures
59:15of uh the researchers. What are
59:18researchers rewarded for? So going into
59:20the question of what they are rewarded
59:22for uh leads us to a deeper
59:26understanding of why sometimes science
59:29can uh move astray can can go astray
59:34into problematic uh domains.
59:39For a long time it has been assumed that
59:43traditional uh metrics are uh rewarding
59:48quantity rather than quality. So for
59:50people who are not very familiar with
59:52research metrics and how research is
59:54assessed.
59:56Uh important times researchers produce a
59:59large number of articles.
1:00:01The difficulty is to assess the quality
1:00:04of the science included in these
1:00:07articles. So what is the quality of
1:00:10research that I have produced in my
1:00:12career in order to reward me for that to
1:00:15give incentives to do even better
1:00:18research and so on and so forth.
1:00:20It is a very difficult problem to
1:00:22consider especially if somebody has
1:00:25produced a large number of articles
1:00:28because uh of the time necessary to read
1:00:32in depth all those articles and to make
1:00:35strong assessments about them. In most
1:00:39situations, it's very unrealistic to
1:00:41believe that in every assessment uh
1:00:44scenario, for instance, when I apply for
1:00:46a research position, when I apply for
1:00:49promotion, where I apply for some uh
1:00:53reward
1:00:55that the uh committees will be able to
1:00:58read all of my research. That's where
1:01:01the so-called research metrics play a
1:01:03role. Those are automatic tools that use
1:01:07uh online uh material and make a summary
1:01:12of the quality of research that I
1:01:15produce. For instance, uh age index
1:01:18impact factor citations. So it's
1:01:21publicly available. How many other
1:01:23works, how many other uh scientific
1:01:27articles have referred to my own work?
1:01:30This is called citation. So with metrics
1:01:33one can see immediately how many
1:01:34articles that a researcher has, how many
1:01:38citations they have and other more
1:01:40concrete and more specialized uh uh
1:01:44metrics such as the age index and the
1:01:46impact factor. So the age index is a is
1:01:50an effort to make sure that uh somebody
1:01:53has a good a critical mass of important
1:01:57work. at the same time many uh studies
1:02:01and at the same time impactful too
1:02:04rather than having a very large number
1:02:06of uh works that nobody has ever uh
1:02:09cited uh before or that they have a
1:02:13unique lack if you wish uh
1:02:18run that has received thousands of
1:02:20citations but they haven't really
1:02:22followed up with with additional
1:02:24research anyway. So these are specific
1:02:27technical uh uh
1:02:31uh metrics
1:02:33but the problem with them is that uh
1:02:36they can be gained. In other words, they
1:02:40are used as proxies to the quality of
1:02:42research. But after they become public,
1:02:45we know
1:02:46people spo stop responding
1:02:50uh in the
1:02:52impassionate manner of just doing my
1:02:54best to produce my research and then
1:02:57these metrics will be just captured that
1:02:59I will be conscious of the metrics
1:03:01beforehand before I conduct my research
1:03:03and I will adjust my practices. All of
1:03:06us uh basically uh will adjust our
1:03:09practices in such a way to uh
1:03:13to be rewarded to be positively assessed
1:03:18using these methods. So this is famously
1:03:21captured by the concept that when a
1:03:23measure becomes a target, it ceases to
1:03:25be good. Instead of trying to produce
1:03:27the best possible research, we try to
1:03:29achieve the best possible metrics.
1:03:34So the uh movement for the responsible
1:03:37use of metrics criticizes specific types
1:03:40of uh uh metrics that they are very
1:03:43shallow that they don't show the a
1:03:46well-rounded uh picture of the of a
1:03:50researchers's careers and therefore
1:03:52essentially that they distort incentive
1:03:54for researchers often times resulting to
1:03:57the problems uh their reproducibility
1:03:59problems that we mentioned before.
1:04:03So responsible metrics it has been
1:04:05argued must be adjusted for context uh
1:04:10specific discipline, career stage,
1:04:13research type and the way that they
1:04:16should be used is in in a not in a naive
1:04:19way but in a more sophisticated way such
1:04:22as using multiple indicators
1:04:25uh combined with qualitative assessments
1:04:28and so on and so forth. So from the
1:04:31junior researchers point of view, these
1:04:34comments have been a little bit uh above
1:04:38your head if you wish because they they
1:04:40speak a little bit from the perspective
1:04:42of the the research assessor of
1:04:45established researchers. But when you
1:04:47don't have your h your research yet, uh
1:04:51how should you why should you be
1:04:53interested into that as as a startup
1:04:55researcher?
1:05:00For instance, I will clarify this now
1:05:02and I will return. You should be ready
1:05:05to present yourselves to present your
1:05:08work in the form of a
1:05:11a new approach if you wish. It is called
1:05:14the narrative CV. So as you as you grow
1:05:17in your scientific journey, you will be
1:05:20having a CV that has a
1:05:23a fixed form. Most notably, it will have
1:05:27a list of publications. Okay? And then
1:05:29you will have most likely a scholar
1:05:31Google profile which will show also your
1:05:34uh your citations.
1:05:36The important thing and how this should
1:05:38concern uh junior researchers is to keep
1:05:41in mind that you should also have at the
1:05:45same time an Arabic C.
1:05:48Generally speaking, how I view an
1:05:50article a bit is that it it describes
1:05:53your research journey. Okay? Every time
1:05:57that you are attracted to a research
1:05:59problem and an agenda, there is a little
1:06:01bit of a story behind that. Why why have
1:06:04you been interested in that research
1:06:06question and then why were you drawn in
1:06:08the other projects and so on and so
1:06:10forth? How do you make all of your books
1:06:14consistent? So essentially
1:06:18uh what were what were your
1:06:19contributions in the
1:06:23you know in any given uh research
1:06:25project? What were you why were you
1:06:27attracted to that? What do you think you
1:06:29have learned from this exercise in your
1:06:32own personal tone rather than just uh
1:06:35presenting a shallow bullet point set of
1:06:39bullet points of individual uh
1:06:41publications?
1:06:45So the narrative simply contextualizes
1:06:48the descriptions, emphasizes quality,
1:06:50emphasizes the personal aspect, the
1:06:53personal contribution within a research
1:06:55pro uh a project, the exact role in the
1:06:58team, why you were attracted into that,
1:07:02why did you expect this to achieve terms
1:07:04of social societal impact? Okay. And uh
1:07:09other aspects such as mentoring and
1:07:11public engagement. Uh so once more with
1:07:15the new tools from AI uh it is
1:07:18increasingly uh low cost to produce high
1:07:21quality uh uh material and at the same
1:07:27time of course adjusted to your exact
1:07:29personal uh research journey and
1:07:32prospect and uh and perspective excuse
1:07:36me of uh you know how you made your
1:07:39choices, what were your contributions,
1:07:41how do you think are the uh the
1:07:44important uh implications of your of
1:07:47your research and so on and so forth. So
1:07:50again for junior researchers should be
1:07:53already focusing on uh more qualitative
1:07:56presentations of your work at the same
1:07:59times those should be combined
1:08:02before
1:08:03uh traditional quantitative metrics. For
1:08:07instance, we have a recent paper where
1:08:11you have three possible candidates that
1:08:13are caused by example one, example two
1:08:15and example three. Instead of just
1:08:17checking the uh if you wish the the
1:08:20number of their citations or the number
1:08:22of their publications, you see a much
1:08:24more advanced picture using a large
1:08:27number of uh indices to account for
1:08:30different uh uh parts of their
1:08:34productivity, their career, their impact
1:08:37to their discipline and maybe of
1:08:40possible possibilities to capture
1:08:42problematic uh practices. Okay, we don't
1:08:45have enough time to do that but we can
1:08:46see that there are not just one metrics
1:08:48but one metric but very many different
1:08:51metrics that can be carefully used
1:08:54combined uh with uh a narrative story if
1:08:59necessary in order to really be able to
1:09:02properly assess uh researchers.
1:09:06So now I will sum up this uh this part
1:09:10by speaking about the dor and the
1:09:13principles.
1:09:15Those were the key initiatives that
1:09:18argue favor of uh
1:09:22responsible research metrics.
1:09:24Okay. So they are closely related to the
1:09:29uh movements in within Europe. For
1:09:32instance, uh UK is also one part of
1:09:35them. the the effort to be uh consistent
1:09:39with the principles of these initiatives
1:09:41and declarations.
1:09:44So basically, roughly speaking, they
1:09:46just uh uh both of them argue in favor
1:09:50of a more holistic and well-rounded
1:09:53approach to research evaluation, not
1:09:55only shallowly based on uh a
1:09:58quantitative indicators taking into
1:10:01account of diverse contributions,
1:10:03uh social impact uh and so on and so
1:10:07forth.
1:10:16Okay. So we have the uh
1:10:20the narrative situation. What's it
1:10:22included more or less is what we
1:10:24discussed uh discussed
1:10:27discussed before. uh the specific
1:10:30contributions within a uh within a team,
1:10:33the implications of the
1:10:35of the project, potentially
1:10:39uh potential methodological innovations,
1:10:41impact and so on and so forth, as well
1:10:44as uh team building and mentoring and
1:10:48your role in this uh uh in this
1:10:51engagement.
1:10:55I think we might be running a little bit
1:10:58uh low in time. Uh
1:11:02how much time do we have left?
1:11:06>> Five more minutes.
1:11:10>> Okay, great.
1:11:13So I will uh summarize the discussion
1:11:16with the recent identity integrity
1:11:19aspects. Apologies and uh I will leave
1:11:22the last part with the interact
1:11:25interactive question for uh sorry
1:11:29interactive uh session for uh
1:11:32considering when we share the uh this
1:11:35lecture notes. Uh so another important
1:11:40aspect that the junior researchers uh uh
1:11:44should know are all the aspects of
1:11:47research integrity. For instance, when I
1:11:50was conducting my first uh experiments
1:11:54uh with human participants, I uh I had
1:11:58to take an online course on the
1:12:00implications of uh uh
1:12:04uh
1:12:06implications of running experiments with
1:12:08with human subjects. the ethical
1:12:09principles that should be respected all
1:12:12the components that should be carefully
1:12:14followed in order to make this uh
1:12:17consistent with ethical principles and
1:12:19foundations.
1:12:22So you should all be familiar with the
1:12:25core principles of research integrity
1:12:27such as honesty, objectivity,
1:12:30transparency, accountability, fairness
1:12:33and respect.
1:12:36Okay. And in particular when this goes
1:12:40into personal interaction with
1:12:42participants
1:12:44who will generate uh data for scientific
1:12:48purposes. To this one should argue uh
1:12:52informed consent a respect for the
1:12:56personality and respect for the
1:12:58principle of non harm. So whenever there
1:13:02are interventions with human
1:13:03participants, we should make sure that
1:13:05they the benefits that they receive are
1:13:07always uh uh greater than the costs.
1:13:13But what should be important I mean this
1:13:16should shall not be an exclusive list
1:13:18for you. What should it be important for
1:13:20senior researchers is to make sure that
1:13:23they address uh aspects of integrity and
1:13:27uh uh within the scientific education at
1:13:31an early stage which I think still needs
1:13:35to be uh improved upon and for junior
1:13:39researchers try to get as much
1:13:42information as possible about these
1:13:44aspects to avoid the fact that later on
1:13:47in your uh research career you might end
1:13:49up conducting empirical studies or
1:13:52experiments and you find yourself a
1:13:55little bit immature regarding how to
1:13:58protect uh ethical principles and you
1:14:01know this might complicate things. So
1:14:03it's much much more important in the
1:14:05first place from the beginning to be
1:14:08clear about how you design your research
1:14:10in order to respect these principles.
1:14:13So finally what we should be uh very
1:14:17careful is to avoid question of research
1:14:19practices.
1:14:21The first category is pure fraud uh
1:14:24fabrication of plagiarism. There are
1:14:27very few scientists practically almost
1:14:30nobody goes into science with the uh
1:14:33objective of uh
1:14:36presenting false data. However this
1:14:40happens but very frequently. But what is
1:14:42catch for frequent is what we have been
1:14:44discussing before hacking uh hacking
1:14:49selective reporting and so on and so
1:14:51forth. These fall into the category of
1:14:53question of research practices
1:14:55uh sometimes one can be also held
1:14:59accountable for problematic practices
1:15:01there. Sometimes it is an issue of gray
1:15:04zone but uh as especially as young
1:15:08researchers just make sure that you
1:15:11don't do any of those practices. You use
1:15:13pre-registration, you use open science
1:15:16to exclude that you use this kind of
1:15:19problematic practices that previous
1:15:21generations were subject to meta
1:15:25research. Uh this might be the most
1:15:28important uh uh contribution of meta
1:15:31research. it goes to the front and shows
1:15:33how you can avoid uh doing all those
1:15:36problematic practices.
1:15:42So this is just an example. This is just
1:15:45an example and I will sum up with this
1:15:48of uh gray areas and difficult
1:15:50decisions. When is it a decision that
1:15:54must be
1:15:55followed versus
1:15:59one
1:16:01that should be avoided on the basis of
1:16:03ethical principles.
1:16:05How should you deal with outliers
1:16:08from the one perspective? One number
1:16:10might be very extreme in your data
1:16:14uh which make things unrealistic and
1:16:17biased. So it should be removed. On the
1:16:20other hand, if you're dealing with
1:16:22outliers who have not been to specified,
1:16:25for instance, in a British registration
1:16:27document, there concerns
1:16:30to what degree will the removal of
1:16:32outliers be uh
1:16:37self-erving.
1:16:38So for instance when you observe the
1:16:40data with and without the outlier what
1:16:43kind of temptations will be to uh have
1:16:47flexibility what's keep
1:16:53registration
1:16:54uh what kind of how much mile what kind
1:16:57of exploratory analysis should we have
1:16:59done before the registration this is
1:17:01still an open question and a gray area
1:17:05uh despite the progress that have been
1:17:10Authorship still remains uh an issue.
1:17:13Who has contributed enough? In some
1:17:15domains, uh lab managers are uh part of
1:17:20a publication
1:17:22uh for everything that is produced with
1:17:26within their lab. Should this be the
1:17:29case or not? Again, this is a little bit
1:17:31of a of a difficult uh decision.
1:17:34uh
1:17:38how how do you deal with the
1:17:40relationship with your supervisors and
1:17:43partners? This is extremely difficult.
1:17:45What do you do in cases where you uh you
1:17:49see that the practices followed by uh
1:17:52supervisors might might be problematic.
1:17:54This is extremely difficult especially
1:17:57given the importance of a supervisor in
1:18:01uh uh in your career and in your
1:18:05development. Then that emphasizes
1:18:07particularly for junior people the
1:18:09importance of uh choosing uh supervisors
1:18:13in a uh you know in the mo in the most
1:18:16mature possible way and having access to
1:18:20uh the people who will be able to help
1:18:24in in terms of this uh these great areas
1:18:27in terms of research. So I think I think
1:18:30this is it. We're out of time. There
1:18:34there are some uh interesting particular
1:18:37scenarios that we are that we are
1:18:39considering. These can be left as food
1:18:41for thought for uh for the participants.
1:18:45Uh
1:18:47and these are the key uh takeaways from
1:18:50uh today's presentation. It's been a
1:18:53real pleasure. Apologies for not being
1:18:55there and being able to deliver it uh
1:18:58live. Uh so I don't know if we might
1:19:02have a little bit of uh time for
1:19:03questions. Thank you.
1:19:06>> Thank you very much Professor Madavis
1:19:08for accepting to do it even though you
1:19:11could not be physically here. Do you
1:19:13have any questions for Professor
1:19:15Madavis? There's nothing from the online
1:19:17attendees.
1:19:20We will make sure that you we share all
1:19:23your contact details together with your
1:19:25presentation. So uh later in future if
1:19:28someone wants to contact you for uh
1:19:31anything further uh they will. Thank you
1:19:34very much for your insight. Thank you
1:19:36very much uh for being with us and have
1:19:39a nice day ahead. Thank you.
1:19:42>> Thank you. Thank you so much.