pequeno texto com sugestões construtivas para a FCT, apresentadas no âmbito da Reunião na SEC, com Painel Peritos, em Lisboa, Março 2015

 The Honourable Schoolboy

An Appraisal of FCT (with constructive suggestions)

Paulo Vargas Moniz

FCT has reached a maturity level (within several levels it has previously attained and others it will eventually) and a strategy appraisal is now needed. My academic\scientific life has ‘accompanied’ FCT, (actually even when JNICT (the pre or proto FCT) was in place) and now, as full professor and vice rector for research, I hope I can be of assistance and bear a meaningful and positive contribution (having lived and been part of other research institutes on different parts of the world). The following notes should not be seen as criticisms (they are not!) but rather as constructive and tentative starting points for a broad discussion, if considered adequate.

On top, I would suggest that FCT could become more emphatically a twofold respected and well embraced (by the Portuguese scientific community) agency. On the one hand, FCT has to be the agency to evaluate and rank research (centres, units, labs as well as projects) and, on the other hand, FCT also has to be  the agency to fund research. I know this seems just repeating what some will say is the current status, but I do mean becoming a well accepted and respected agency for those aims. Because some loss of trust and empathy has taken place (in recent years) and FCT is not seen as a (welcome) partner any more. I bring no faults and oriented accusations; I just share a broad and common feeling as the recent changes in methodology and evaluation have broken some working ties*.

*I have also been the coordinator for the RAD (lecturers evaluation, ranging from the administration up to the final decisions by the Rector) hence I am well aware of ‘dangerous’ is evaluating; I am also editor and referee, teacher...

When I point to FCT to become the agency to evaluate and rank research, I take A3ES (an agency that evaluates and ranks higher education teaching in Portugal) as a useful comparing line. A3ES has a methodology that FCT could henceforth adapt (if and where applicable): A3ES  involves former Rectors, former and current lecturers (maybe some could constitute better  ‘top notch’ examples, I concede, so that their words would carry convincing examples by means of prior demonstrated actions…), students, evaluators from other (mainly EU) countries. All those that are evaluated are visited in situ, there are different steps towards reaching a recommendation. Sometimes not to the liking of a university (like closing a PhD course for lack of quality) but on the whole A3ES is fully respected. FCT will need to recover or get to that, so to convey the challenges in the EU-2020 initiatives for research and innovation. In other words, A3ES is seen as a participant of value, a member of a wider ‘team’, a team of the community, if not a leader in the process (maybe mainly administratively) and FCT is now surely not that. I repeat, this is not a criticism. It is just a conclusion after many innuendos and perhaps some lack of political “savviness” on either participants in the recent evaluation process of research units as well as re-framing the way grants are allocated. There is a bit of a sort of  secessionist attitude, more in the sense of  “institutions go along but just’’ rather than breaking away from the system. But this is like a convoy of ships in a stormy sea where neither new navigation maps (accepted by all, even if with different reference frames and units for orientation) nor an efficient (possibly shared tasks of) leadership is observed and is mostly absent. My suggestion: look at A3ES, adapt, and not necessarily proceed to improvise (blindly); Re-structure FCT governance method, mostly how it interacts with the HE institutions because it has not been working well (recently).

When I point to FCT to become the agency to fund research I must be clearer. Funding is most of the times the crucible in research. So I appeal for FCT to adopt in the near-mid future some approach, whereby:

• -Had FCT allowed in situ visits by the evaluation panels  (i.e., effectively allowing  those research groups into the 2nd round or phase of evaluation) to those research centres  that had scored 14\20 (maybe with some additional criteria) would had provided an invaluable  ‘emotional’ stimuli in those research units. It is more than the fact of getting eventually (as they did) funds for re-structuring. It was the opportunity to show and clarify, and that represented a 1 point (in 20) of allowed uncertainty as a sort of tolerance gap to eventually assess, to possibly compensate possible ‘’errors’’ by the panels. Politically, I considered an error not having done so, in spite of widespread suggestions: the costs of putting some more visits would have been far more than compensated by the involvement and chance to participate and reducing a priori any potential protestation, that followed, with the CRUP memorandum asking for that 14\20 allowance effectively ignored. This came to a heavy price of “confrontation” and was not needed. More concretely, (some) research units are really not motivated at all for the challenges [laid down by the SEC –Secretary of State of Science -  in an important talk in December, in ISEG, Lisbon 2014]; Researchers just take the ‘business as usual’ line, no need to change (much), money will eventually come (from new government promises), no need to struggle or change  (so much). This is a dangerous route and Portugal may fail by a large margin the aims of e.g., getting 2/3 of investment in research from private industry, e.g., or getting PhD graduates into enterprises. The description on the ‘lab corridors’ is that (i) no one knows how to achieve that, (ii) there is no leadership to explain how to achieve that and (iii) can research groups trust a leadership where some funds were made available  (into the restructuring fund) but where no political dialogue exists between separate parts and not merging at all? A sort of political vacuum or governmental scientific leadership is being claimed (It is unfair, I know, and incorrect, but this what has been said). Research units are composed by human beings and some 5-year  plan soviet-style is felt to have been imposed (ignoring the need to motivate ‘troops’). It is not working, it will not work. Before proceeding, I do not want my words to be meant as personal criticisms. They are not; they are merely  sharing what I collect in my several functions.

• -Assure teams of evaluators of projects are senior and above ‘temptations’

• -Better scheduling of incoming grants, programmes of projects funding, preferably with one year or so in advance

• -Research projects and some type of grants should be subjected to a ‘pre-selection’ with eventual  ‘re-teaming’. Let me be more concrete. When resources are scarce and only 1 project per objectives is to be funded, where actually 2 or 3 teams of equal worth are in the competition, all submitting very similar projects, why set them up against each other in the evaluation (to then discard some, then launching appeals, time consuming negative discussions, waste of public money on long decision processes) when a panel or pre-evaluation could suggest (if appropriate, of course) teams to join in a single project, sharing funds? I am aware competition is not necessarily bad but when resources are scarce and drying a source to some teams is equivalent to end it (when all are of equal worth, assuming).... FCT needs to be (more) constructive, politically savvy, repairing communication lines, leadership and trust bonds in a pro-active and broadly constructive manner…

• -FCT should provide better access to bibliometry tools for the institutions e.g., supporting a broad national subscription of (some, not necessarily all) SciVal [Elsevier]  modules. It is only fair to ask for better productivity if one provides the means to evaluate it, beyond the usual or obvious that everybody can use to count (e.g. mere scholarly output without fine tuned filters for specific areas). Please have a look at SciVal. There are more than citation counts per journals and 1st or 2nd quarter of quality rankings. FCT wants better and more productivity? It can do more and better than just funding. FCT can provide tools too for assistance and self-filtering within each institution. Just deciding on  research funds by means of a panel appraisal is not enough: more is needed in terms of providing tools for institutions to self-measure them and discuss objectively i.e. do a rational a self-assessment.

• -In addition, FCT should be sensitive and savvy to provide funds because sometimes  the funds that a team X with N elements requests, another team Y with M members, where N=2M, can do the same for half the cost (and has proven so)  (I am aware the arithmetic’s suggest this immediately but nevertheless funds go to X and none to Y). Hence, FCT could be showing some leadership on project selection on the grounds of useful bibliometric data and observables, to ensure better and proper use of public funds.

• -FCT must also assure that technicians that assist projects execution are up to the task, including in terms of ‘emotional intelligence’. From a very small number of technicians (just a few but even a few causing discomforts) there are sometimes no willingness to become part of the solution, too. I.e., these technicians are willing to assist i.e. they provide information but they stop short of assisting in looking on how to solve and where to look for it. Researchers have pointed that there is a sort of grudge (really) in some technicians towards PhD holders.

The following are more focused and narrowed suggestions to concrete situations, that aim to get better refereeing, inc.  better matching between projects and evaluation; FCT could eventually adopt:

• Every PI (Principal Researcher) , for each project, besides keywords, will have to insert 4 to 6 items of classification (if possible) from e.g. PACS (http://publish.aps.org/PACS), MSC (http://www.ams.org/mathscinet/msc/msc2010.html), or equivalent, wherever available or appropriate.
• Current and new referees will also have (i.e., mandatory) to provide:
• Expertise described in terms of classification scheme PACS, MSC, e.g., used in his\her papers
• Provide as well a  list of up 10 his\her papers, in the past, up to 20 years, where those papers were indeed within those classification scheme  elements e.g. PASC, MSC:
• Nb. Papers in some areas include those classification scheme elements, when printed and some top journals also request them in the submission process
• It is not enough for a referee to say 'I am an expert on this'; This assertion NEEDS to be verified and can be  so by means of the classification scheme and the papers cited, matched, demonstrated
• Whenever possible, the matching between referees and projects will be under the elements of classification scheme above, which will take priority above keywords
• This must be checked within an informatics process as herein suggested; i.e., check the referees' papers PACS or MSC  or equivalent (inserted by means of editorial labeling); do not assume that what the referees indicated  as 'expertise' is satisfactory… [Some referees have indeed been poorly related experts to some proposals..]

• FCT should use a more basic and easy to 'make a decision' ranking, like most top journals use at the editorial level of decision:
• E.g., use the CQG (http://iopscience.iop.org/0264-9381) or PRD (http://prd.aps.org/) ranking of papers, suitably adapted for projects 
• This means 5 levels, from 1. [top 10%],  i.e. sort of quotas..., 2. [high quality, international competitive], 3. [average...] 4. [not yet suitable but can improve] 5. [reject]
•  Funds will only be given to those ranked 1.; for those ranked 2, only if funds remain.
• Rank 3 (and marginally rank 4) may only be re-appraised if e.g. severe budget cuts and changes are re-introduced in the proposals (FCT being proactive...; this will be better elaborated in the following)
• Evaluators must subject all proposals in rank 2, 3 (and possibly 4) to go through as many tests as required, so that only very few would eventually remain in rank 3 or 4, with the bulk either in rank 1, some in 2., the rest ‘sent’ into rank 5.

•  FCT should provide more stringent criteria for the referees to use; E.g.:
• Did the  PI & team published on the project area in e.g. the  last 5 or 10 years?
• Were those papers of quality? Can the referees actually read and understand through those papers? No? Then ask an 'expert'... To appraise the quality, referees ought to evaluate more than just quantity\metric indicators
•  And the team, is it a good assembled team? Can they cooperate? Have they cooperated before? Is the team 'artificial'?
• Can the research proposed still be conducted (give or take one year) without that all requested budget (resources asked, material or human)? 

Politically:

1. FCT should have regular meetings with a CRUP (Portuguese Universities Rector’s Council) Commission on research; CRUP should also have more and better coordination between members in terms of research (policy and internal governance, joint) I mention this in the wake of talks with counterparts who are in agreement that a voice or via a CRUP sector could be heard in the FCT, with respect to applications and requests on a given subject.

2. FCT has Scientific Committees but are chosen by 'selection' which is for 'a kind of contest' without having a voice (in the selection) by CRUP. What I mean is that someone in the FCT choose from the 'established' who will be in the scientific committees. And in these there is not, I repeat there is not  a good geographical representation at the level of the HEIs and what they do; HEIs are producing the output that justifies the existence of the FCT as funding agency.

3. In the light of national and EU directives for a change of paradigm in research conduction, meaning involving gradually much more involvement and investment from and into industry, FCT should also have a committee from the industry , i.e. listen internally to it, even by having some vice President to the task or guiding research evaluation and funding within better and more innovation, transfer of knowledge. The FCT of the past decade is NOT enough, as it is, a different FCT, not necessarily larger or shorter , but different. The paradigm is different, the challenge is different (and considerable)

4. Operationally, FCT has not been teaming with  the intrinsic agendas of Univs (CRUP). What I mean is that HEIs have calendars (eg, academic year and semester) and rules to be complied with eg A3ES. But the FCT does  NOT coordinate with CRUP via a committee or committees for this, imposes instead an agenda for projects, competitions and is sometimes not sensible at all;

5. There is a lack a serious articulation of FCT and CRUP on bi partnerships and multi lateral programmes; FCT does not articulate the bilateral with 'A.Integradas' .

6. 5. FCT could also first consult the CRUP wrt regulations and implementing grants and projects. I mention this because the composition of processes, registration, submission and then review (peer) collides with the agendas of the HEI (see 3. above). FCT technical rarely provides explanations on time.

7. FCT should have a channel \ interlocutors with CRUP \ HEI See above 1 .; I mean that the FCT should also have it someone who had the task of listening, dialogue with the CRUP \ HEI. I do not want to use the term 'ombudsman' but there should be an executive, serious, responsible, which had been provided management experience \ academic governance at a high level (a  former (vice)Rector, eg) and therefore serve as direct advice to the FCT  ( one 'special' adviser on  regulations and procedural rules)