text:
Journal topic
Hydrol. Earth Syst. Sci., 23, 557–572, 2019
https://doi.org/10.5194/hess-23-557-2019
Hydrol. Earth Syst. Sci., 23, 557–572, 2019
https://doi.org/10.5194/hess-23-557-2019

Research article 30 Jan 2019

Research article | 30 Jan 2019

# Assessment of food trade impacts on water, food, and land security in the MENA region

Assessment of food trade impacts on water, food, and land security in the MENA region
Sang-Hyun Lee1, Rabi H. Mohtar2,3, and Seung-Hwan Yoo4 Sang-Hyun Lee et al.
• 1Research Institute for Humanity and Nature (RIHN), Motoyama 457-4, Kamigamo, Kita-ku, Kyoto 603-8047, Japan
• 2Department of Biological and Agricultural Engineering, Texas A & M University, College Station, USA
• 3Faculty of Agricultural and Food Sciences, American University of Beirut, Beirut, Lebanon
• 4Department of Rural and Biosystems Engineering, Chonnam National University, Gwangju, Republic of Korea

Correspondence: Rabi H. Mohtar (mohtar@tamu.edu, mohtar@aub.edu.lb)

Abstract

The Middle East and North Africa (MENA) region has the largest water deficit in the world. It also has the least food self-sufficiency. Increasing food imports and decreasing domestic food production can contribute to water savings and hence to increased water security. However, increased domestic food production is a better way to achieve food security, even if irrigation demands an increase in accordance with projected climate changes. Accordingly, the trade-off between food security and the savings of water and land through food trade is considered to be a significant factor for resource management, especially in the MENA region. Therefore, the aim of this study is to analyze the impact of food trade on food security and water–land savings in the MENA region. We concluded that the MENA region saved significant amounts of national water and land based on the import of four major crops, namely, barley, maize, rice, and wheat, within the period from 2000 to 2012, even if the food self-sufficiency is still at a low level. For example, Egypt imported 8.3 million t yr−1 of wheat that led to 7.5 billion m3 of irrigation water and 1.3 million ha of land savings. In addition, we estimated the virtual water trade (VWT) that refers to the trade of water embedded in food products and analyzed the structure of VWT in the MENA region using degree and eigenvector centralities. The study revealed that the MENA region focused more on increasing the volume of virtual water imported during the period 2006–2012, yet little attention was paid to the expansion of connections with country exporters based on the VWT network analysis.

1 Introduction

Food security and water scarcity are urgent socioeconomic and environmental issues in the Middle East and North Africa (MENA) region (Saladini et al., 2018), which are highly interlinked, and a water–energy–food (WEF) nexus has been suggested as a proper and integrated approach for resource management (Bazilian et al., 2011; Rasul, 2014; Mohtar and Daher, 2014; Lee et al., 2018). For example, food security in the MENA region has become complicated due to increased risks owing to the geopolitical challenges and inability to satisfy needs with domestic production because of the lack of adequate arable land and water resources (Rastoin and Cheriet, 2010). In addition, food imbalance in the MENA region is forecast to reach 60 % in 2050 and food security in the MENA region could be extremely compromised (Rastoin and Cheriet, 2010). Climate change could lead to more frequent occurrence of extreme climatic events in Mediterranean region, accompanying a 50 % decrease in agricultural production by the end of the century (Porter et al., 2014). In particular, water saving through food trade can be suggested as a solution for mitigating groundwater depletion in the MENA region (Lezzaik et al., 2018).

Previous studies showed that the effective import of virtual water may reduce water use for domestic food production in importing countries and help alleviate water stress in the MENA region, where the largest water deficit in the world exists (Gleick, 2000; World Bank, 2009). The critical condition of water scarcity in the MENA region will reach severe levels by 2025 (Tolba and Saab, 2009). In addition, if population increases rapidly and urbanization continues fast, availability of water could be reduced in the Arab countries by approximately 50 % by the year 2025 (Abahussain et al., 2002). Water shortages will certainly speed up the rate of desertification in the Arab countries (Abahussain et al., 2002). Agricultural water withdrawals account for over 85 % of the total water withdrawn by the various countries of the MENA region (FAO, 2014). Irrigation systems in the MENA region are based on pumping groundwater resources, such as aquifers, and water security is being threatened by the declining aquifer levels and the extraction of nonrenewable groundwater (Antonelli et al., 2015). In addition, Immerzeel et al. (2011) expected that the unfulfilled water demand in the entire MENA region would increase from the current level of 16 % to 51 % in 2040–2050 owing to climate changes. The zone of severely reduced rainfall extends throughout the Mediterranean region and the northern Sahara (Hennessy et al., 2007). Milly et al. (2005) estimated that climate change will cause a decrease in water runoff by 20 % to 30 % in most of the MENA region by 2050, mainly owing to the rising temperatures and lower precipitation. In addition, the regions that include Syria, Lebanon, Israel, and Jordan will get drier, with significant rainfall decreases in the wet season.

However, the high dependency on food import can be a risk of food security, even if it can elicit domestic water, energy, and land savings in water-scarce regions. Therefore, we should consider a trade-off between food security and resource savings using a holistic approach, such as a trade–WFL (water–food–land) nexus. Furthermore, the VWT can be suggested as relevant to the water policy of a nation (Schyns and Hoekstra, 2014), thus establishing a new point of view from which both food security and sustainable water management are considered (Novo et al., 2009).

This study addresses three questions that relate to the role and impact of the VWT in the MENA region, which are raised to draw attention to the complexity of the issue and the need for a broader view in assessment. Specifically, (1) what are the effects of the VWT on water savings and land tenure in the MENA region? (2) Has the structure of the virtual water import in the MENA region been vulnerable or robust? (3) Who are the influential importers and exporters in the VWT network in the MENA region?

The aim of this study is to evaluate the effects on water savings and land tenure from importing crops in 15 countries in the MENA region such as Algeria, Egypt, Iraq, Jordan, Kuwait, Lebanon, Libya, Morocco, Oman, Qatar, Saudi Arabia, Syria, Tunisia, the UAE, and Yemen. In addition, we quantified the amount of VWT from 2000 to 2012, and we analyzed the structure of the VWT, such as the connectivity and influence in the MENA region using degree and eigenvector centralities.

2 Materials and methods

## 2.1 VWT based on international trade

The VWT represents the water embedded in international trade, and it indicates the water used in the exporting country to produce crops for export. Therefore, the VWT is calculated based on the water footprint of exporters, which indicates the total amount of water used for producing crop, and the export of virtual water in the exporting country has the same meaning as the import of virtual water has in the importing country. For example, Saudi Arabia imported wheat from various exporters, and the virtual water import (or export) was calculated by multiplying the quantity of traded wheat with the respective water footprint of exporters. Accordingly, the main factors for quantifying a VWT are the trade data and water footprint, and the VWT is calculated by multiplying the trade by its associated water footprint in the exporting country, as follows:

$\begin{array}{}\text{(1)}& \mathrm{VWT}\left[{n}_{\mathrm{e}},{n}_{\mathrm{i}},c,t\right]=\mathrm{CT}\left[{n}_{\mathrm{e}},{n}_{\mathrm{i}},c,t\right]×\mathrm{WFP}\left[{n}_{\mathrm{e}},c\right],\end{array}$

where the variable VWT denotes the virtual water trade from the exporting country, ne, to the importing country, ni, in year t, as a result of trade in crop c; CT represents the crop trade from the exporting country, ne, to the importing country, ni, in year t as a result of trade in crop c; and WFP represents the water footprint of crop c in the exporting country, ne.

The international trade data of the four major crops, namely, barley, maize, rice, and wheat from 2000 to 2012, were obtained from FAOSTAT (http://www.fao.org/faostat/, last access: 15 July 2018), as shown in Table 1. The crop with the largest amount of import was wheat, with 27.6 million t yr−1 imported by the MENA region from 2000 to 2012, followed by maize (14.4 million t yr−1), barley (9.0 million t yr−1), and rice (3.7 million t yr−1).

Water footprint is a localized index for countries, accounting for the climate, productivity, and irrigation. In this study, we considered water footprints of all countries in the world; however, a lot of effort would be required to estimate the water footprints of all countries and this was outside the scope of the current study. Therefore, we applied water footprint data of 147 countries, including those in the MENA region, from the study executed by Mekonnen and Hoekstra (2010). The water footprint for a crop is divided into green and blue water footprints based on the water resources (Hoekstra and Chapagain, 2008). The green water footprint indicates that water supplied by precipitation is retained in the soil of the root zone (Falkenmark, 1995), and blue water footprint is the water stored at the surface or in the ground. Therefore, the green water footprint is related to rain-fed agriculture and the blue water footprint is related to irrigation water provided by aquifers or surface bodies of water. Water saving could be also divided into green and blue water savings through water footprints.

## 2.2 Water and land savings due to international food trade in an importing country

Food import is also related to domestic water and land savings. In particular water saving has a different meaning from virtual water import. For example, Saudi Arabia imported wheat from various exporters and virtual water import indicates the sum of the products obtained from multiplying the quantity of imported wheat by the respective water footprint of each exporter. However, water saving indicates the amount of water needed to produce the same quantity of imported products domestically. Therefore, water saving due to wheat import in Saudi Arabia is estimated by multiplying the quantity of imported wheat with the water footprint of wheat in Saudi Arabia.

In this study, we applied green and blue water footprints of crops in each country in the MENA region, as shown in Table 1. However, the availability of water footprint data in the MENA region was limited in some cases. For example, the water footprint of wheat was available in all countries except for Bahrain. Land savings have the same implication as water savings; thus we calculated land savings using the land footprint of each country in the MENA region, as shown in Table 2. The land footprint indicates the land requirement for producing 1 t of crops, and it was calculated based on the harvest area and crop production data collected from FAOSTAT (Table 1).

Table 1Cultivation area, production, and the quantity of crops imported in the MENA region from 2000 to 2012.

a Average value from 2000 to 2012 provided from FAOSTAT (http://www.fao.org/faostat/, last access: 10 January 2019); b average value from 2000 to 2012 provided from World Bank (https://data.worldbank.org/, last access: 10 January 2019).

The water and land savings could be assess the impacts of the failure of trade on domestic water and land requirements in the importing country. Although this assumption about water and land savings considers an extreme trade situation, these results could be used to understand the importance of the international crop trade in the MENA region. In other words, the water and land savings indicated the amount of water and land requirements for crops imported to substitute domestic production, and the water and land savings were calculated as follows:

$\begin{array}{}\text{(2)}& & \mathrm{WFP}\left[{n}_{\mathrm{i}},c\right]=\frac{\mathrm{CWR}\left[{n}_{\mathrm{i}},c\right]}{P\left[{n}_{\mathrm{i}},c\right]},\text{(3)}& & \mathrm{LFP}\left[{n}_{\mathrm{i}},c\right]=\frac{\mathrm{Area}\left[{n}_{\mathrm{i}},c\right]}{P\left[{n}_{\mathrm{i}},c\right]},\text{(4)}& & \mathrm{WS}\left[{n}_{\mathrm{i}},c\right]=\mathrm{CI}\left[{n}_{\mathrm{i}},c\right]×\mathrm{WFP}\left[{n}_{\mathrm{i}},c\right],\text{(5)}& & \mathrm{LS}\left[{n}_{\mathrm{i}},c\right]=\mathrm{CI}\left[{n}_{\mathrm{i}},c\right]×\mathrm{LWP}\left[{n}_{\mathrm{i}},c\right],\end{array}$

in which variable WFP [ni, c] (m3 t−1) is the water footprint of crop c in the importing country ni, CWR is the crop water requirement (m3), and P is the production (t). Equivalently, LFP [ni, c] (ha t−1) is the land footprint of crop c in the importing country ni, and “Area” is the cultivated area (ha). The symbol WS (m3) or LS (ha) indicates the amount of water or land savings in the importing country ni. CI is the import of crop c in the importing country ni.

## 2.3 Degree and eigenvector centralities for analyzing the structure of VWT

### 2.3.1 Nonscaled and scaled in-degree centralities of VWT

Understanding the VWT structure is important for quantifying the amount of import and export because the VWT structure can represent whether it would be sustainable or vulnerable. For example, if a country imports considerable amounts of virtual water through the food trade from just a few exporters, the structure of VWT in this country might be easily influenced by exporters. However, if a country is connected with many exporters in VWT, it can have a resilient structure for global changes. A few studies have been conducted on the analysis of the structure of the VWT using a network-based approach (Konar et al., 2012; Dalin et al., 2012; Lee et al., 2016). For example, Konar et al. (2012) analyzed the characteristics of the network change in virtual water trade and found that a number of export trade partners followed an exponential distribution in 2000. Dalin et al. (2012) found that constant organizational features were observed in the network of VWT even though the number of trade connections and the volume of VWT has been growing. In addition, Lee et al. (2016) analyzed the vulnerability of the importing countries through the characteristics of the network in VWT.

Table 2Water and land footprints of four major crops in the MENA region.

a Water footprint data were referenced by Mekonnen and Hoekstra (2010); b land footprint was calculated by crop production and cultivated area provided from World Bank open data (https://data.worldbank.org/, last access: 15 July 2018).

In this study, we analyzed the links of the VWT network for identifying the VWT structure using degree centrality, that is, the number of degree incidents on a given node (Freeman, 1979). In addition, the degree centrality is divided into in- and out-degree centralities, depending on the direction. “In-degree” is based on the number of lines (or volume) directed to the node and “out-degree” is based on the number of lines (or volume) that the node directs to. A node indicates the country in the global trade network, and incidents mean the trade between countries which can be amounts of products or number of connections; for example, if one country exports product to five countries, that country has five incidents. In this study, we focused on the in-degree centrality because the MENA region includes representative importing countries. An importer accompanying an increased in-degree centrality has expanded connectivity with a large number of exporters, meaning that this importer could cope with an accidental disconnection from a certain exporter. In addition, the volume of products exported or imported can be applied to incidents as the weight of links. In this study, the in-degree centrality, based on the VWT network, is expressed according to the nonscaled in-degree centrality (NSInDC), that is, based on the number of links, and the scaled in-degree centrality (SInDC), that is, based on the volume of links.

$\begin{array}{}\text{(6)}& & {\mathrm{NSInDC}}_{i}=\sum _{j}^{N}{\mathrm{Link}}_{ij}/\left(N-\mathrm{1}\right),\text{(7)}& & {\mathrm{SInDC}}_{i}=\sum _{j}^{N}{\mathrm{Flow}}_{ij}/\left(N-\mathrm{1}\right),\end{array}$

where NSInDCi is the nonscaled in-degree centrality of country i, and Linkij is the number of links between the ith and jth countries. The symbol SInDCi is the scaled in-degree centrality of country i, and Flowij is the volume of virtual water traded between the ith and jth countries. Moreover, N is the total number of countries that trade with a given MENA country.

Through NSInDC and SInDC, we analyzed the vulnerable expansion (or reduction) and robust expansion (or reduction) in the VWT network in the MENA region. For example, the vulnerable expansion in the network indicates that the amount of flow to a node increases but the number of connections to other nodes decreases. This is represented by high levels of SInDC and low levels of NSInDC. The importer country that is associated with vulnerable expansion has an increased quantity of products from only a few exporters.

### 2.3.2 Eigenvector centralities of VWT

In general, connections to nodes which are themselves influential could make a node more influential than connections to less influential nodes (Newman, 2016), and eigenvector centrality can be used for measuring the influential connections (Ruhnau, 2000). For example, the concept of eigenvector centrality has been used by the web search engine Google in order to rank web pages (Berry and Browne, 2005; Bryan and Leise, 2006; Newman, 2016).

Table 3The annual water and land savings based on imported crops in the MENA region from 2000 to 2012.

* Water and land savings by rice import were not calculated because of the lack of data of water and land footprints in the MENA region.

In the VWT network, the eigenvector centrality could be used for identifying influential countries that could affect the entire network. In other words, the entire VWT can be affected by a few influential countries, and it is important to identify these countries for understanding and estimating the change in the entire structure of the VWT. An eigenvector centrality can measure the influence of each country in the entire VWT, and it is related not only to its own connection pattern but also to the connections of other countries to it. Therefore, a country is more influential if it is considered in relation to the countries that are influential themselves (Ruhnau, 2000). The eigenvector centrality assigns relative centrality to all of the countries in the VWT, based on the principle that connections to high-level centrality countries contribute more to the centrality of the countries compared to equal connections to low-level centrality countries (Ruhnau, 2000; Lee et al., 2016). Bonacich (1972) defined the centrality (xi) of a node i as the positive multiple of the sum of adjacent centralities in links (or volume) between nodes (Aij). Therefore, if we denote the centrality of vertex i by xi , then we can allow for this effect by making xi proportional to the average of the centralities of i's network neighbors (Newman, 2016),

$\begin{array}{}\text{(8)}& {x}_{i}=\frac{\mathrm{1}}{\mathit{\lambda }}\sum _{j=\mathrm{1}}^{n}{A}_{ij}{x}_{j},\end{array}$

where λ is a constant. Defining the vector of centralities x=(x1, x2, …), we can rewrite this equation in matrix form as

$\begin{array}{}\text{(9)}& \mathit{\lambda }\mathbit{x}=\mathbf{A}\mathbit{x}.\end{array}$

This type of equation is solved using eigenvalues and eigenvectors, where A  is an adjacency matrix of Aij, and λ is a scalar, known as the eigenvalue associated with the eigenvector c defined as a column vector. Eigenvector centrality is determined by calculating the principal eigenvector that has the largest eigenvalue among all eigenvectors. A nonnegative eigenvector with the maximal eigenvalue exists. We refer to a nonnegative eigenvector (x≥0) of the maximal eigenvalue as the principal eigenvector, and we call the entry xi the eigenvector centrality of node (country) i (Ruhnau, 2000).

Table 4The amounts of green and blue water imported in the MENA region from 2000 to 2012.

3 Results and discussion

## 3.1 Trade-offs between national water–land savings and food security through food trade in the MENA region

This study considered trade-offs between food security and food trade in terms of national resource management. For example, increasing domestic food production instead of importing food products could be one policy for food security, but additional water and land for domestic products would need to be considered at the same time. In other words, food imports could contribute to domestic water and land management; therefore, we estimated the national water and land savings by importing crops as shown in Table 3. In Saudi Arabia, blue water savings by barley, maize, and wheat imports were estimated to be 5.0, 2.0 and 0.8 billion m3 yr−1, respectively. In comparison to the internal water resource of Saudi Arabia, which is 2.4 billion m3 yr−1 as shown Table 1, the water saving through the import of barley, maize, and wheat could be considered to be significant in Saudi Arabia. In the case of Egypt, most of the water saving occurred based on the imports of wheat and maize. Approximately 7.5 billion m3 yr−1 of blue water was saved by importing wheat. Specifically, the internal water resources in Egypt are only 1.8 billion m3 yr−1 (Table 1); therefore, water scarcity could be an issue for food security policy in Egypt. Lebanon was strongly influenced by the impact of crop import on land savings. Approximately 0.24 million ha could be saved by crop imports, comprising 36 % of the agricultural area in Lebanon, which indicates that the crop trade in Lebanon has significant benefits in terms of land resources compared to water resources.

Food imports could be regarded as a negative factor in food security, and it is obvious that food security would accompany water and lands for domestic food products. These results showed that food imports could result in positive impacts on numerous water and land savings in the MENA region. However, there are limitations to these results. First, water saving estimated in this study was based on the hypothetical situation that meat there was no international trade situation, and sometimes the water saving estimate was larger than the internal water resources in some countries, such as Saudi Arabia and Egypt. Additionally, some crops are required for the specific type of climate but this study assumed that MENA region was suitable for cultivating maize, wheat, barley, and rice.

Figure 1The total amount of virtual water imported by each country in the MENA region from 2000 to 2012, separated into green (a) and blue (b) water.

## 3.2 The VWT in the MENA region from 2000 to 2012

### 3.2.1 Virtual water import in the MENA region

The total amount of green and blue water imported by each MENA country from 2000 to 2012, respectively, reached 921.2 and 80.5 billion m3 in the MENA region, as shown in Table 4 and Fig. 1. The largest volume of green water was imported annually by Egypt (19.1 billion m3 yr−1), followed by Saudi Arabia (11.9 billion m3 yr−1). In addition, the largest amount of blue water was imported annually by Saudi Arabia (1.2 billion m3 yr−1), followed by the UAE (0.9 billion m3 yr−1). Over 70 % of the green water imported annually into the MENA region based on the trade of barley (approximately 8.5 billion m3 yr−1) was occupied by Saudi Arabia. The amount of virtual water imported based on the trade of maize was 13.0 billion m3 yr−1, with Egypt being the primary importer of 31 % of the total imported amount into the MENA region.

Generally, rice is cultivated in paddy fields, and the blue water footprint of rice in these fields is larger than other cereal crops in various countries. For example, the global average of the blue water footprint of rice is 584 m3 t−1 but that for wheat is 343 m3 t−1 (Chapagain and Hoekstra, 2011; Mekonnen and Hoekstra, 2010). Therefore, the importers of rice also import a lot of water. Approximately 3.0 billion m3 yr−1 of blue water was imported in the rice trade from 2000 to 2012, and Saudi Arabia, the UAE, and Iraq were the primary importers. The largest volume of virtual water imported by the MENA region was due to the trade of wheat. The annual amount of virtual water imported based on the trade of wheat in the MENA region from 2000 to 2012 was approximately 42.6 billion m3 yr−1, and over 35 % of the virtual water imported through the wheat trade was imported by Egypt (15.7 billion m3 yr−1). However, the amount of blue water was only 2.0 billion m3 yr−1 because the green water footprint is much larger than the blue water footprint in main exporters such the Russian Federation, Australia, and Canada, which might indicate that wheat has been cultivated in rain-fed areas with less irrigation.

Figure 2Virtual water imported per capita in the MENA region from 2000 to 2012.

We also estimated the amount of virtual water imported per capita (VWIcap), as shown in Fig. 2, which shows the differing viewpoints regarding food and water securities. If we consider only the total amount of imported virtual water, the UAE may not be considered to be a significant importer because the population and area of the UAE is much smaller than those of the other countries in the MENA region, such as Saudi Arabia. However, the virtual water import per capita in the UAE is larger than that of Saudi Arabia, thus indicating that the dependency on virtual water imported from exporters in the UAE is much more significant than in Saudi Arabia. For example, the VWIcap was 1266.6 m3 cap−1 yr−1 in the UAE, which was the largest value in the MENA region. The UAE is strongly dependent on the import of virtual water, even though the UAE imports only 4.2 billion m3 yr−1 of virtual water. The VWIcap increased significantly in Saudi Arabia and Libya from 2000 to 2012. For example, Saudi Arabia imported approximately 314 m3 cap−1 yr−1 of virtual water in 2000 and it increased to 768 m3 cap−1 yr−1 in 2012. Saudi Arabia was the second largest importer in the MENA region, and its VWIcap was also the fifth highest in the MENA region.

### 3.2.2 Virtual water export to the MENA region

We also focused on the volume of virtual water exported to the MENA region by each exporter from 2000 to 2012, as shown in Fig. 3. Based on the trade of barley, Ukraine exported 41.1 billion m3 of green water to the MENA region that amounted to 27 % of the total green water imported in the MENA region. In terms of blue water traded through barley, five exporters (Germany, Australia, the Russian Federation, Ukraine, and India) provided 78 % of the total blue water imported in the MENA region based on barley. Based on the trade of maize, Argentina contributed 40 % of the total amount of green water imported by the MENA region based on maize, but the blue water imported by the MENA region was primarily from the USA. Based on the trade of rice, the major virtual water exporters to the MENA region were India, Thailand, and Pakistan. In particular, 30.4 billion m3 of blue water was imported from these countries from 2000 to 2012, which comprised 78 % of the blue water imported by the MENA region based on rice. Wheat was the most representative crop imported by the MENA region. The Russian Federation and the USA provided 25 % (140.6 billion m3) and 21 % (111.2 billion m3) of the total amount of green water imported in the MENA region based on the trade of wheat in 2000 to 2012, respectively, and the remaining 55 % was divided among several exporters, including Australia, Canada, France, and Ukraine.

## 3.3 The temporal change in VWT structure in the MENA region

From 2000 to 2012, both the volume and connectivity of VWT was changed. For example, the virtual water imported in the MENA region slightly increased and the VWT was distributed with more exporters in 2006, as shown in Fig. 4. However, the volume of virtual water imported in the MENA region increased more than 50 % from 2006 to 2012 but the distribution of VWT seemed to be consistent. In the case of Lebanon, VWT was strongly dependent on the USA, Argentina, and Australia. However, Lebanon expended the VWT in 2006 and the Russian Federation, Turkey, and Kazakhstan contributed to virtual water imports in Lebanon, as shown in Fig. 4. Accordingly, the structure of VWT in Lebanon approached a distributed network. However, the VWT in 2012 showed that it was dominated by Ukraine and the Russian Federation, though Lebanon imported more virtual water in 2012 than 2006.

Figure 3Quantities of green water export (GWE) and blue water export (BWE) from the primary exporters to the MENA region from 2000 to 2012.

These changes are more related to the structure of VWT and the MENA region should consider not only the amount of virtual water but also the structure of VWT for sustainable food security, subject to the condition of a strong dependency on crop import. Therefore, we analyzed the degree centralities of NSInDC and SInDC from 2000 to 2012 in the MENA region and identified the countries who had the vulnerable expansion or reduction in the VWT network. Figure 5 shows the NSInDC and SInDC patterns in the VWT network in accordance with each country in the MENA region. If a specific country has both large NSInDC and small SInDC, this country has connections with various exporters but imports a small amount of virtual water. Specifically, Egypt and Yemen showed that NSInDC was lower but SInDC was higher than other countries, thus indicating the intensive connectivity with a few exporters. In contrast, Saudi Arabia had larger SInDC than other countries expect for Egypt, while the NSInDC was also the highest of the MENA region. Accordingly, Saudi Arabia had a more distributed structure regarding VWT. The UAE and Iraq had similar SInDC in 2012 but NSInDC was quite different – 0.46 for the UAE and 0.27 for Iraq. Furthermore, SInDC in Morocco (96.45) was larger than in the UAE (83.41) but NSInDC in Morocco (0.26) was smaller than in the UAE (0.46). In comparison to the UAE, Morocco had intensive connections with fewer exporters compared to the UAE.

Figure 4Virtual water imports at the MENA region and Lebanon in 2000, 2006, and 2012. Others indicate the countries which export less than 100 Mm3 yr−1 to the MENA region or Lebanon.

Based on the temporal changes in NSInDC and the SInDC during two periods (2000–2006 and 2006–2012), the MENA region countries were divided into four types (I–IV), as shown in Fig. 6. The x axis indicates the NSInDC and the y axis indicates the SInDC. Type I countries are located at higher levels both in the x and y axis and show a robust expansion in the virtual water import. Additionally, the countries in this type increased the connectivity and volume of virtual water imported, simultaneously. Type II countries increased the volume of virtual water imported without expansion of connectivity. Type III countries showed reductions in the virtual water import with reduction of connectivity, and type IV countries have established connections with more exporters but have decreased virtual water imports.

Figure 5Nonscaled and scaled in-degree centralities of each country in the MENA region in 2000, 2006, and 2012.

In the early 2000s, most countries in the MENA region expanded their trade structure by increasing both the connectivity to the exporters and the volume of the imported virtual water. In Bahrain, Omen, Qatar, Yemen, Saudi Arabia, Lebanon, and the UAE, the NSInDC of the VWT network increased significantly from 2000 to 2006, which means that the trade connectivity expanded. The expanded structure of the VWT indicates that the MENA countries were connected to various exporters and that this structure can be a resilient structure for global changes. In particular, the import of food crops is an essential factor in food security in the MENA region, even if food self-sufficiency is increased by increasing domestic production. However, Egypt had the largest SInDC but NSInDC was ranked sixth among the MENA region countries. In 2006, Egypt and Saudi Arabia both expanded the connectivity in the VWT network, as shown by the increasing NSInDC. However, the type of VWT structure in many MENA countries such as Yemen, Qatar, Bahrain, and Lebanon has moved to Type II, which means that the countries increased the volume of the imported virtual water, but the number of exporters that linked to the MENA countries decreased from 2006 to 2012. In particular, in 2012, most countries kept their connectivity or reduced it, except for Algeria, Iraq, Libya, and the UAE. These results indicate that the dependence of the MENA region on virtual water import increased rapidly recently with the large increase in the imported volume of virtual water. However, the connectivity of the VWT in the MENA region has not increased as much as the volume of virtual water imported increased.

The degree centrality in this study could be useful for identifying the connectivity and volume of trade of each country, but it is limited to show the influence of each country on entire trade network; thus we estimated eigenvector centrality, as shown on Fig. 7. In 2000, Egypt and Saudi Arabia were identified as the most influential importers in the MENA region, and the USA and Australia were the most influential exporters. Accordingly, the entire VWT in the MENA region could be affected by these importers and exporters. This means that the change in the trade policy or food management in these countries could change the structure of VWT in the MENA region. In 2006 and 2012, the influential countries in the MENA region were still Egypt and Saudi Arabia, but the influential exporters moved to the Russian Federation, Ukraine, and Brazil.

Figure 6Country types in the MENA region according to the changes in nonscaled and scaled in-degree centralities.

## 3.4 Importance and limitations of water footprint and VWT in the MENA region from a policy perspective

Generally, the VWT is more related to resource management in exporting countries rather than importing countries because the embedded water in food trade indicates water resources that are consumed for producing food products in the exporting country. However, VWT is also considered to be an important issue in importing countries in terms of water and food security. For example, the reduction of VWT might be related to water consumption by replacing imported food products by domestic food products.

Figure 7Eigenvector centralities of the virtual water trade network in the MENA region in 2000, 2006, and 2012.

However, the application of the concept of VWT is under critical discussion (Wichelns, 2010). First, water footprints formulate new concepts of water management, but we need to realize that water footprint can be changed due to various factors such water requirement, productivity, production system, development of technologies, fertilizer usage, and irrigation scheduling and operations of the water facilities. Second, VWT could contribute to the connection of water management to food security. However, food trade is affected by the scarcity or affluence of other important resources, such as capital, labor, and land (Biewald et al., 2014). In particular, economic values, such as the price of food products, are the main driver in global food trade, but there is no global value established for virtual water. Therefore, it is difficult to apply virtual water to trade policy in terms of the economic efficiency. Therefore, policy makers or resource managers in the MENA region should not only consider the effects of VWT but also the difficulty in adapting virtual water to policies for resource management. Third, there are spatial and temporal issues of VWT in the study. The VWT could be affected by geopolitical issues such as topography and distances between importers and exporters. For example, the changes in exporting countries in the MENA region could be related to energy use for transporting products; thus trade policy should consider the economic benefit or cost of transportation. Therefore, the VWT should be discussed with geopolitical issues such as benefit and cost of transportation. In addition, VWT and water–land savings by food trade in this study were calculated based on a historical database; thus it was difficult to apply the results to future policy.

Despite these limitations, we believe that virtual water has a role in the achievement of sustainable water, land, and food security, even if there are limitations and difficulties in applying the virtual water concept. As mentioned above, the VWT can be a major resource in the MENA region. Accordingly, vulnerable VWT, for example, low connectivity, can be a risk element for future food security risk management. In particular, the MENA region is strongly dependent on food products from exporting countries, which implies a strong dependency on water resource from exporting countries. Therefore, water shortages or low-food production in exporting countries might cause increasing food prices in the MENA region, but also increasing domestic water use for increasing domestic food production. The primary resources of water, energy, and food are naturally interlinked. The degree of their interlinkages in the MENA region is exceptionally high, thus creating a higher degree of risks and vulnerability. Therefore, understanding these interlinkages and quantifying them in an attempt to better understand this complex system of systems is crucial. This requires the synergistic effort of multiple disciplines, including contributions from various technologies, science, policies, health, communication, and economics, at local-process- and system-level scales. In this study, we believe that the VWT in the MENA region can be the key factor for bridging water and food, and it is important to quantify the influence of trade on water and food management. In addition, this study revealed vulnerable (or robust) expansion (or reduction) and influential traders in the VWT network in the MENA region, based on in-degree and eigenvector centrality indices. If a country in the MENA region has low connectivity but an increased import of virtual water, this country should re-evaluate their vulnerable trade structure and change their trade policy or water–food management.

4 Conclusions

The import of water in virtual form based on VWT could develop into a major water portfolio that dominates water management in the water-scarce countries of the MENA region. In water-deficit areas, such as the MENA region, the VWT can offer new perspectives for understanding and solving water stress and scarcity. In summary, this study showed that the significant water in comparison to internal water resources could be saved by food trade in the MENA region, and policy makers can benefit by considering both the quantitative impacts of VWT and the structural changes in VWT, such as vulnerable expansion (or reduction) in the MENA region. For example, when a country in the MENA region sets a plan for increasing food security, this country first should identify the amount of water and land savings that can be achieved by food import and consider the trade-off between food security and food import. In addition, the stable trade could be a component for stable food supply in the MENA region; thus this study contributes to the understanding of the dependency on each trade partner for countries in the MENA region and can help with setting the food trade policy in terms of extension (or reduction) of trade partners and increase (or decrease) in volume of trade.

However, this study only focused on food trade and water–land savings; thus the energy part was not considered. The MENA region represents an extreme case globally in terms of water and energy resources; for example, 66 % of the world's known crude oil reserves, but only 1.4 % of the world's fresh water supplies is attributed to the region (Khater, 2001). The increase or decrease in water withdrawal for irrigation is related to the energy used for water extraction such as pumping surface or groundwater. For example, 5 % or more of the total electricity consumption can be attributed to water pumping in Saudi Arabia (Siddiqi and Anadon, 2011). Energy use for food production and water supply could be the main factor in integrated resource management in the MENA region, and the lack of the energy part was a limitation in this study.

In spite of this limitation, the intensity and connectivity of VWT, which were analyzed in this study, can be the major components needed for integrating resource management in the MENA region. Accordingly, VWT is regarded as the important factor in determining food security and water–land management, and it can be a useful interlinking parameter among resources in the WEF nexus approach, which identifies key issues in food, water, and energy securities through the lens of sustainability, seeking to predict and protect against future risks and resource insecurities (Biggs et al., 2015). The core of the nexus concept is that the production, consumption, and distribution of water, energy, and food are inextricably interlinked; thus this study would provide important information to policy makers for evaluating scenarios about integrated resource management toward sustainability in the MENA region.

Data availability
Data availability.

The international trade data, cultivation area, and production of study crops (barley, wheat, maize, and rice) are available via FAOSTAT (http://www.fao.org/faostat/, last access: 10 January 2019), and water footprint data are available via the Water Footprint Network (https://waterfootprint.org, last access: 10 January 2019). The results data for this study are freely available by contacting the corresponding author.

Author contributions
Author contributions.

S-HL and RHM conceived and designed the research; S-HL and S-HY analyzed the data; S-HL contributed analysis tools; and S-HL and RHM wrote the paper.

Competing interests
Competing interests.

The authors declare that they have no conflict of interest.

Acknowledgements
Acknowledgements.

This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2018R1A6A3A03010868).

Reviewed by: Marc Walther and Trent W. Biggs

References

Abahussain, A. A., Abdu, A. S., Al-Zubari, W. K., El-Deen, N. A., and Abdul-Raheem, M.: Desertification in the Arab Region: analysis of current status and trends, J. Arid Environ., 51, 521–545, 2002.

Aldaya, M. M., Allan, J. A., and Hoekstra, A. Y.: Strategic importance of green water in international crop trade, Ecol. Econo., 69, 887–894, 2010.

Allan, J.: Fortunately there are substitutes for water otherwise our hydro-political futures would be impossible, in: Priorities for water resources allocation and management, ODA, London, 13–26, 1993.

Antonelli, M., Laio, F., and Tamea, S.: Food security and VWT in the Middle East and North Africa, Int. J. Water Resour. Dev., 31, 326–342, 2015.

Bazilian, M., Rogner, H., Howells, M., Hermann, S., Arent, D., Gielen, D., Steduto, P., Mueller, A., Komor, P., Tol, R. S. J., and, Yumkella, K. K.: Considering the energy, water and food nexus: Towards an integrated modelling approach, Energ. Policy, 39, 7896–7906, 2011.

Berry, M. W. and Browne, M.: Understanding search engines: mathematical modeling and text retrieval, SIAM, 17, 84–88, 2005.

Biewald, A., Rolinski, S., Camoen, H. L., Schmitz, C., and Dietrich, J. P.: Valuing the impact of trade on local blue water, Ecol. Econ., 101, 43–53, 2014.

Biggs, E. M., Bruce, E., Boruff, B., Duncan, J. M., Horsley, J., Pauli, N., McNeill, K., Neef, A., Van Ogtrop, F., Curnow, J., Haworth, B., Duce, S., and Imanari, Y.: Sustainable development and the water–energy–food nexus: A perspective on livelihoods, Environ. Sci. Policy, 54, 389–397, 2015.

Bonacich, P.: Factoring and weighting approaches to status scores and clique identification, J. Math. Sociol., 2, 113–120, 1972.

Bryan, K. and Leise, T.: The \$25,000,000,000 eigenvector: The linear algebra behind Google, SIAM Rev., 48, 569–581, 2006.

Chapagain, A. K. and Hoekstra, A. Y.: The blue, green and grey water footprint of rice from production and consumption perspectives, Ecol. Econ., 70, 749–758, 2011.

Chapagain, A. K., Hoekstra, A. Y., and Savenije, H. H. G.: Water saving through international trade of agricultural products, Hydrol. Earth Syst. Sci., 10, 455–468, https://doi.org/10.5194/hess-10-455-2006, 2006.

Dalin, C., Konar, M., Hanasaki, N., Rinaldo, A., and Rodriguez-Iturbe, I.: Evolution of the global VWT network, P. Natl. Acad. Sci. USA, 109, 5989–5994, 2012.

Fader, M., Gerten, D., Thammer, M., Heinke, J., Lotze-Campen, H., Lucht, W., and Cramer, W.: Internal and external green-blue agricultural water footprints of nations, and related water and land savings through trade, Hydrol. Earth Syst. Sci., 15, 1641–1660, https://doi.org/10.5194/hess-15-1641-2011, 2011.

Falkenmark, M.: Land-water linkages: a synopsis. Land and Water integration and river basin management, FAO Land Water Bull., 1, 15–16, 1995.

Falkenmark, M. and Lannerstad, M.: Food security in water-short countries – Coping with carrying capacity overshoot, Re-thinking Water and Food Security: Fourth Botin Foundation Water Workshop, CRC Press/Taylor & Francis, Boca Raton, USA, 3–22, 2010.

FAO – Food and Agriculture Organization of the United Nations: on-line database, available at: http://www.fao.org/nr/water/aquastat/main/index.stm (last access: 10 January 2019), 2014.

Freeman, L. C.: Centrality in social network: conceptual clarification, Soc. Networks, 1, 215–239, 1979.

Gleick, P. H.: The world's water 2000–2001, The biennial report on freshwater resources, Island Press, Washington, D.C., USA, 19–38, 2000.

Hanjra, M. and Qureshi, M.: Global water crisis and future food security in an era of climate change, Food Policy, 35, 365–377, 2010.

Hennessy, K. B., Fitzharris, B., Bates, B. C., Harvey, N., Howden, M., Hughes, L., Salinger, J., and Warrick, R.: Australia and New Zealand: climate change 2007: impacts, adaptation and vulnerability: contribution of Working Group II to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change, Cambridge University Press, Cambridge, UK, 2007.

Hoekstra, A. Y.: VWT: Proceedings of the international expert meeting on VWT, Value of Water Research Series No. 12, UNESCO-IHE, Delft, the Netherlands, 2003.

Hoekstra, A. Y. and Chapagain, A. K.: Globalisation of Water: Sharing the Planet's Freshwater Resources, Wiley-Blackwell Publishing, New Jersey, USA, 2008.

Hoekstra, A. Y. and Hung, P. Q.: Globalisation of water resources: international virtual water flows in relation to crop trade, Global Environ. Change, 15, 45–56, 2005.

Immerzeel, W., Droogers, P., Terink, W., Hoogeveen, J., Hellegers, P., Bierkens, M., and van Beek, R.: Middle-East and Northern Africa water outlook, World Bank Study, Future Water Report, Future Water, Wageningen, the Netherlands, 1–136, 2011.

Khater, A. R.: Intensive groundwater use in the Middle East and North Africa, Intensive use of groundwater: challenges and opportunities, CRC Press, Boca Raton, USA, 355–386, 2001.

Konar, M., Dalin, C., Hanasaki, N., Rinaldo, A., and Rodriguez-Iturbe, I.: Temporal dynamics of blue and green VWT networks, Water Resour. Res., 48, W07509, https://doi.org/10.1029/2012WR011959, 2012.

Lee, S.-H., Mohtar, R. H., Choi, J.-Y., and Yoo, S.-H.: Analysis of the characteristics of the global virtual water trade network using degree and eigenvector centrality, with a focus on food and feed crops, Hydrol. Earth Syst. Sci., 20, 4223–4235, https://doi.org/10.5194/hess-20-4223-2016, 2016.

Lee, S. H., Taniguchi, M., Mohtar, R., Choi, J. Y., and Yoo, S. H.: An Analysis of the Water-Energy-Food-Land Requirements and CO2 Emissions for Food Security of Rice in Japan, Sustainability, 10, 3354, https://doi.org/10.3390/su10093354, 2018.

Lezzaik, K., Milewski, A., and Mullen, J.: The groundwater risk index: Development and application in the Middle East and North Africa region, Sci. Total Environ., 628, 1149–1164, 2018.

Mekonnen, M. M. and Hoekstra, A. Y.: The green, blue and grey water footprint of crops and derived crop products, Value of Water Research Series No. 47, UNESCO-IHE, Delft, the Netherlands, 2010.

Milly, P. C., Dunne, K. A., and Vecchia, A. V.: Global pattern of trends in streamflow and water availability in a changing climate, Nature, 438, 347–350, 2005.

Mohtar, R. H. and Daher, B.: A platform for trade-off analysis and resource allocation: the water-energy-food nexus tool and its application to Qatar's food security [part of the `Valuing Vital Resources in the Gulf' series], Chatham House, London, UK, 2014.

Molden, D.: Water for food, water for life, Earthscan, London, and International Water Management Institute, Colombo, 2007.

Newman, M. E.: Mathematics of networks, The new Palgrave dictionary of economics, Palgrave Macmillan, London, UK, 1–8, 2016.

Novo, P., Garrido, A., and Varela-Ortega, C.: Are virtual water “flows” in Spanish grain trade consistent with relative water scarcity?, Ecol. Econ., 68, 1454–1464, 2009.

Oki, T. and Kanae, S.: VWT and water resource, Water Sci. Technol., 49, 203–209, 2004.

Porter, J. R., Xie, L., Challinor, A. J., Cochrane, K., Howden, S. M., Iqbal, M. M., Lobell, D. B., and Travasso, M. I.: Food Security and Food Production Systems, in: Contribution of Working Group II to the Fifth Assessment Report of the IPCC, Cambridge University Press, Cambridge, UK, 485–533, 2014.

Rastoin, J. L. and Cheriet, F.: Food security in the Mediterranean: a major geostrategic issue, Les Notes d'Ipemed, Etudes & Analyses, Ipemed, Paris, 2010.

Rasul, G.: Food, water, and energy security in South Asia: a nexus perspective from the Hindu Kush Himalayan region, Environ. Sci. Policy, 39, 35–48, 2014.

Ruhnau, B.: Eigenvector-centrality: a node-centrality?, Soc. Networks, 22, 357–365, 2000.

Saladini, F., Betti, G., Ferragina, E., Bouraoui, F., Cupertino, S., Canitano, G., Gigliotti, M., Autino, A., Pulselli, F. M., Riccaboni, A., Bidoglio, G., and Bastianoni, S.: Linking the water-energy-food nexus and sustainable development indicators for the Mediterranean region, Ecol. Indicat., 91, 689–697, 2018.

Schyns, J. F. and Hoekstra, A. Y.: The Added Value of Water Footprint Assessment for National Water Policy: A Case Study for Morocco, Plos ONE, 9, e99705, https://doi.org/10.1371/journal.pone.0099705, 2014.

Siddiqi, A. and Anadon, L. D.: The water–energy nexus in Middle East and North Africa, Energ. Policy, 39, 4529–4540, 2011.

Tolba, M. K. and Saab, N. W.: Arab environment: Climate change, Arab Forum for Environment and Development, Beirut, 2009.

Wichelns, D.: Virtual water: A helpful perspective, but not a sufficient policy criterion, Water Resour. Manage., 24, 2203–2219, 2010.

World Bank: Water in the MENA region: Management Perspectives and Innovations, edited by: Jagannathan, N. V., Mohamed, A. S., and Kremer, A., World Bank, Washington, D.C., 2009.

Yang, H., Wang, L., Abbaspour, K. C., and Zehnder, A. J. B.: Virtual water trade: an assessment of water use efficiency in the international food trade, Hydrol. Earth Syst. Sci., 10, 443–454, https://doi.org/10.5194/hess-10-443-2006, 2006.


text:
# Working with quadratic Lie algebras

A quadratic Lie algebra is a Lie algebra with an invariant inner product and the main examples are semisimple Lie algebras. This definition then makes sense in any linear symmetric monoidal category. I have a series of questions about generalising constructions from semisimple Lie algebras to this abstract setting. Instead of one post with a list of questions (which is frowned on) I have in mind a series of posts. This will also allow me to "edit" posts to take into account any responses before posting.

The first construction is the universal enveloping algebra. This is not usually regarded as mysterious but there are some points to ponder. There are actually two constructions one is as a deformation of the symmetric algebra of the adjoint representation. This was Poincare's construction. This is an early example of a universal quantisation. This makes sense in the abstract setting of symmetric monoidal categories. The other approach was taken by Birkoff-Witt (independently, and decades later) and constructs the universal enveloping algebra as a quotient of the tensor algebra of the adjoint representation. This does not make sense in a symmetric monoidal category which is not abelian.

If we assume/impose the condition that the Casimir is non-zero on every non-trivial irreducible representation then I think the category of representations is semisimple. In this case the Birkoff-Witt construction is defined (and agrees with Poincare's approach).

Now we come to the Yangian. This is defined by Drinfeld by a presentation. More specifically as a quotient of the semi-direct product of the universal enveloping algebra and the tensor algebra. I have not seen a proof that this has the properties it should have. My first question is then whether there is anywhere I can find a proof?

The real question then is what is the simplest construction of the Yangian for a general quadratic Lie algebra? I say simplest because I believe the high-powered machinery of universal quantisations does give one construction. My hope would be constructions analogous to one or both of the constructions of the universal enveloping algebra.

-
Bruce, have you looked at Molev's relatively recent book on Yangians for your foundational question? (He is mostly interested in the classical Lie algebra cases, which admit alternative presentations related to twisted Yangians. I am not convinced that there is only one construction of Yangian: Drinfeld famously gave two very different presentations.) For universal enveloping algebra in symmetric monoidal categories, see mathoverflow.net/questions/25020/… – Victor Protsak Jul 2 '10 at 3:35
Thanks Victor. I have added an answer to question 25020. I will see if I can find Molev's book. My concern is that $SL(n)$ and $GL(n)$ are exceptional in the sense that the adjoint representation appears in the symmetric square of the adjoint representation. Most of the work on Yangians relies on this which is not what I want. – Bruce Westbury Jul 2 '10 at 8:07
This book relies on the evaluation homomorphism. The Yangians for orthogonal and symplectic groups are "twisted Yangians" which are not the usual ones. – Bruce Westbury Jul 2 '10 at 9:43

format: plain


text:
## Controlling Naming in Microsoft Teams

How to prevent users from renaming the Teams

Microsft Teams is one of those products that seems to explode when it’s rolled out. Once it goes viral within the organisation, as it seems to in most, then the number of Teams seems to grow rapidly. This can get out of control and messy unless you have a naming convention. It’s possible to restrict who can create Teams. However it’s difficult to stop people from renaming the Team once it’s created.

In this post, we are going to look at controlling the name of the Teams via a script and a ‘master’ file for the naming.

## The Code

Before we do anything lets take a look at the code…….

<#
.SYNOPSIS
Controls the naming of Teams
.DESCRIPTION
Script to monitor the naming of Teams and correcting them if they are changed.
.NOTES
Version:        1.0
Author:         Matt Browne
Creation Date:  Jan/2020
Purpose/Change: Initial script development
Requires: The PowerShell module for Teams (https://docs.microsoft.com/en-us/microsoftteams/teams-powershell-overview)
To Dos: Needs some logging
#>

Import-Module MicrosoftTeams

$masterFile = "C:\Test\Teams.csv" Connect-MicrosoftTeams -Credential$(Get-Credential)

if (Test-Path -Path $masterFile -ErrorAction SilentlyContinue) {$masterTeamList = Import-Csv -Path $masterFile$teams = Get-Team
foreach ($team in$teams) {
if ($masterTeamList.GroupId -notcontains$team.GroupId) {
Write-Output "New Team added to the master list : $($team.DisplayName)"
$team | Select-Object GroupId, DisplayName, Description | Export-Csv -Path$masterFile -Append
} else {
$teamName =$team.displayname
$officalTeamName =$masterTeamList | Where-Object { $_.GroupId -eq$team.GroupId } | Select-Object -ExpandProperty DisplayName
if ($teamName -ne$officalTeamName) {
Set-Team -GroupId $team.GroupId -DisplayName$officalTeamName
Write-Output "Team -> $officalTeamName, corrected from ->$teamName"
}
}
}
} else {
Write-Output "Initialising master file.  This will not rename any Teams........"
$teams = Get-Team$teams | Select-Object GroupId, DisplayName, Description | Export-Csv -Path $masterFile }  ## A bit about the code The script relies on the PowerShell Teams Module. This allows us to list all the Teams and save them to a ‘Master’ file. If you want to change the location of the Master file, change the following line….. $masterFile = "C:\Test\Teams.csv"

This CSV file holds a list of the Teams that currently exist. The important part is it holds the unique identifier for the Team. This means that whatever the name changes to, we can always identify it.

From this, the script will run through all the Teams and change the name to what is in the master file (if it is different).

Note - If you have a large number of Teams the script could take a while to run.

## Running the script

If you want to run the script as a test, ie without it changing anything, then comment out the following line.

Set-Team -GroupId $team.GroupId -DisplayName$officalTeamName

This means that the script will run and have the normal output but won’t make any changes.

When you run the script it will ask for credentials. This needs to be a Teams admin. If you are going to run this on a schedule then it’s worth having a sensible name for the account. This is because it will appear in the Team as the user that has changed the name back.

## The results

Once you run the script it will look through all the Teams. It then renames anything that doesn’t match what is in the Master file.

The result of will be a message in the Teams ‘General’ Channel showing that the name has changed back.

If the name of the Team needs to change for some reason then it needs to be edited in the CSV file. The script will then make the changes the next time it is run.

## Summary

Run the script on a regular basis and you will have a consistent set of Team names. So when the boss asks you “How to stop people renaming Teams” you have an answer.


text:
# How is a public key actually used to encrypt something?

One thing cryptography articles never seem to explain is how the message actually gets encrypted. You get this long-winded lecture on number theory which ends with, Ta Da! and we have a public and private key. Then, they never explain the process of exactly how some 100,000 byte word document gets encrypted using the public key.

Other times the explanations seem downright misleading. For example, in my book Cryptography (by Meyer and Matyas) it says for RSA the ciphertext is the plain text to the power of key. How do you exponentiate a word document? I don't get it.

• – user991 Mar 12 '14 at 1:18
• How do you exponentiate a word document? A word document is but a series of bytes. When doing encryption you don't care about the underlying format, you treat it as a big number, like the letter A is the number 65 in ASCII. Go grab the nearest Word doc in your system and look for file size in bytes in the properties: You'll get the number $n$. If you decide to encrypt it with textbook RSA, you will have a $8n$-bits long number in your hands. – rath Mar 12 '14 at 1:23
• How do you exponentiate a word document? You don't! – fgrieu Mar 13 '14 at 15:40

Rick Demer already wrote the answer in the very first comment, but without explanation: Hybrid encryption.

But since you asked for a real practical example to encrypt your word document, this is how: Your file is on your disc, and it is 100,000 byte large. You can then do:

• First, you start up a random number generator. Preferably you should either have true randomness or at least a cryptographically secure RNG.
• Let the RNG generate a random 256 bit number.
• Start up your AES 256 engine/program/hardware module, and choose the appropriate mode of operation like CBC. Use the random number as key.
• Use AES to encrypt the entire content of the file.
• Get your public key or generate private/public key, depending on what you want to do.
• Encrypt the random number (used as symmetric key) with your asymmetric encryption scheme. In the case of RSA, do not just use the textbook variant but RSA-OAEP or a similar padding scheme.
• Create a new file: Put relevant information in the file header (whatever you want to be accessible without decryption). In the data of the file, write down the encrypted random number (it was encrypted with the public key), and then you just add the entire encrypted file-content (encrypted with AES).
• Optional: You can add delimeters inside your file to make it easier to see where the key ends and where the ciphertext starts. If you want you can also note down which algorithms you actually used to encrypt this specific file.

AES can obviously be replaced with any other symmetric cipher, you can also pick a different mode of operation or use a specific asymmetic encryption scheme to encrypt the random key. If you do so, putting it into the file might help if you ever forget which file was encrypted with which algorithm.

Note on using asymmetric encryption directly

It is also possible to use for instance RSA directly on your $100,000$ byte document. If your RSA key is $1240$ bit long, then you can split your document into $800$ blocks with each block having $1000$ bit (or e.g. $650$ blocks of $\approx 1231$ bits, whatever suits you). Then you can apply RSA directly to each block and be done with it. The problem is that this is extremely slow. The ratio of processed bits per second is magnitudes lower for asymmetric encryption than for symmetric encryption. And then you have the problem, that RSA shouldn't be used in the textbook variant and RSA-OAEP increases the length.

• Thank you. I don't understand why other people have this mental block about actually explaining how this works. I think its because they don't know themselves in many cases. – Tyler Durden Mar 12 '14 at 16:52
• I've added a comment about padding at that point. – tylo Mar 13 '14 at 12:42
• This is great recipe to make... insecure implementation. Don't roll your own or you will end up with mistakes like here: malleable encryption. – axapaxa Dec 7 '16 at 23:27
• @axapaxa There are a lot of possible things that can go wrong. But with straight hybrid encryption using AES and RSA-OAEP malleable encryption is not one of them. And the "don't implement it yourself" usually refers to implementing encryption methods or designing entirely new protocols. Implementing a generally accepted protocol by utilizing proper cryptographic implementations and abiding their assumptions is quite different. – tylo Dec 8 '16 at 11:29
• @tylo please do explain how AES-CBC (which is proposed in this answer) is not malleable in this case. Implementing generally accepted protocol is usually subject to side-channel attacks, therefore one shouldn't implement it on his own. Especially RSA and AES which are both very prone to side-channels. – axapaxa Dec 8 '16 at 12:44

Well, exponentiating a word document is rather easy. As you've said, it's $100,000$ bytes, or $800,000$ bits. This word document can thus be interpreted as a number between $0$ and $2^{800,000}$. Sure this number may be large, but it can be exponentiated.

However, more commonly symmetric encryption is used with a $128$ or $256$ bit key to encrypt the word document, and then public-key encryption is used on that key, as number between $0$ and $2^{256}$. This approach is much faster.

• What do you mean by "symmetric encryption"? Spell it out for me, how is the document encrypted? – Tyler Durden Mar 12 '14 at 2:38
• @TylerDurden A form of encryption where you have a small shared secret (the private key - there is no public key), and use that small shared secret to encrypt any amount of data. Examples of such algorithms include AES, Salsa20, and more historic examples are Blowfish and (3)DES. – orlp Mar 12 '14 at 3:04
• The details of symmetric algorithms are much more complicated than something like RSA, where encryption is "simply" exponentiation. For instance, the ChaCha20 paper describes a function (in section 2) that gets repeatedly applied to the plaintext. – Stephen Touset Mar 12 '14 at 6:33
• What is described in the first paragraph of the answer A) is never done; B) would not work at all with common RSA key size, it would require special ones with at least a 800,001-bit modulus; C) slows decryption (making it about a million times slower than for common RSA key size, assuming 2-primes RSA and classical algorithms); D) would not be secure unless random padding is thrown in (without it, knowing a reference plaintext, it would be possible to guess a plaintext with only a small change). – fgrieu Mar 13 '14 at 12:05
• @fgrieu I was only explaining to the asker how it would be possible to exponentiate a word document, I did not mean to suggest that it's either secure or practical. – orlp Mar 14 '14 at 12:21

This post covers RSA, other algorithsm will likely have similar considerations but the details may be different.

For textbook RSA the "message" is a number. Due to the use of modular arithmetic this number must be smaller than the modulus.

We can obviously take a sufficiently short sequence of bytes and encode it as a number. However textbook RSA has a number of security concerns, most obviously that since it's determinitic an attacker can guess the message and validate wheher their guess was correct but also some more subtule cases.

To avoid these issues a randomised padding scheme is used. There are several such schemes but I belive OEAP is considered the modern standard. OEAP combines the message with a random number and then subjects the combination to an all-or-nothing transform.

That doesn't work for large messages though, a typical RSA key has a 1024-4096 bit modulus. Add some overhead for padding and even with a 4096 bit key you are talking about a message length limit of arround 5000 bytes.

You could split a large mesage into chunks, encrypt them seperately and then chain together the result but this would be space-inefficient due to seperate randomised padding for each chunk. It would also be slow.

So in practice we encrypt the message with a symetric encryption algorithm and a randomly generated key. We then apply the random padding to the key and encrypt the padded key using RSA with they public key.

One thing cryptography articles never seem to explain is how the message actually gets encrypted.

Then, they never explain the process of exactly how some 100,000 byte word document gets encrypted using the public key.

You break up the document into blocks of a particular byte size. Treat each block as a series of bits. That's a number. Feed the number into the crypto algorithm, get a number back out. That's also a series of bits. Now you have turned a series of plaintext blocks into a series of encrypted blocks.

Other times the explanations seem downright misleading. For example, in my book Cryptography (by Meyer and Matyas) it says for RSA the ciphertext is the plain text to the power of key. How do you exponentiate a word document?

If you really want to see how it works in practice then I encourage you to read the source code. For example, download the source code for BouncyCastle and take a look at the GetInputBlockSize, GetOutputBlockSize, ConvertInput, ConvertOutput and ProcessBlock methods in the RSACoreEngine.cs file. This is only 150 lines of pretty easy-to-follow code but it answers all your questions.

As others have mentioned, it is often impractical to use public key crypto on large documents because the math is slow. Usually what you do is choose a symmetric cryptosystem, choose a random key in that cryptosystem, encrypt the document in that cryptosystem, encrypt the symmetric key using the public key. Now the document cannot be decrypted without the symmetric key, and the symmetric key cannot be decryped without the private key.

• This answer is wrong. It suggest to apply the asymmetric algorithm to blocks of plaintext small enough to make the interpretation of that as a number suitable for computation. A) That's practically never done in serious systems. B) That's unsafe unless random padding is added (which goes against this description, because with usual padding schemes, plaintext blocks only become a number after padding); in particular, knowing one plaintext, it becomes trivial to recognize that another plaintext is identical in the first block. C) We have more efficient methods based on hybrid encryption. – fgrieu Mar 13 '14 at 12:18
• @fgrieu: Perhaps you stopped reading before my last paragraph? – Eric Lippert Mar 13 '14 at 14:00
• The last paragraph says of the part that I criticize " it is often impractical ", citing a performance reason; that's very different from saying: doing it is never done in standard applications, including for security reason, which would be more accurate. I criticize: " You break up the document into blocks of a particular byte size. Treat each block as a series of bits. That's a number. Feed the number into the crypto algorithm, get a number back out. That's also a series of bits. Now you have turned a series of plaintext blocks into a series of encrypted blocks. " – fgrieu Mar 13 '14 at 14:37
• @fgrieu: Of course you are right that in practice, no one uses asymmetric crypto for large documents. But let's not forget that the original poster's question shows that they do not understand even how the document can be made into numbers in the first place. We have a beginner here; let's concentrate on the basics rather than getting off into the weeds of all the pragmatic defenses against real-world attacks. That can come later. – Eric Lippert Mar 13 '14 at 14:41
• BouncyCastle and others unify all cryptographic methods into a single do-it-all API capable of enciphering arbitrary data with any algorithm, including unsuitable for that purpose. That's over-engineering in my opinion, for it becomes all too easy to misuse the API to do non-standard, unsafe and inefficient things, including just what I criticize in your answer. A different example occurred in a recent question where Java's API allowed use of a private RSA key with encryption padding, which is unsafe. – fgrieu Mar 13 '14 at 15:07


text:
### Home > AC > Chapter 2 > Lesson 2.1.4 > Problem2-44

2-44.

Write and simplify the algebraic expression shown in each expression mat below.

1. $\left(2x − 3\right) − \left( ? \right)$

1. Which tile pairs make zeros?

1. $−x −\left(x + 2\right) = −2x − 2$


text:
This section showcases how the mlr3pipelines package can be extended to include custom PipeOps. To run the following examples, we will need a Task; we are using the well-known “Iris” task:

library("mlr3")
task$data() ## Species Petal.Length Petal.Width Sepal.Length Sepal.Width ## 1: setosa 1.4 0.2 5.1 3.5 ## 2: setosa 1.4 0.2 4.9 3.0 ## 3: setosa 1.3 0.2 4.7 3.2 ## 4: setosa 1.5 0.2 4.6 3.1 ## 5: setosa 1.4 0.2 5.0 3.6 ## --- ## 146: virginica 5.2 2.3 6.7 3.0 ## 147: virginica 5.0 1.9 6.3 2.5 ## 148: virginica 5.2 2.0 6.5 3.0 ## 149: virginica 5.4 2.3 6.2 3.4 ## 150: virginica 5.1 1.8 5.9 3.0 mlr3pipelines is fundamentally built around R6. When planning to create custom PipeOp objects, it can only help to familiarize yourself with it. In principle, all a PipeOp must do is inherit from the PipeOp R6 class and implement the .train() and .predict() functions. There are, however, several auxiliary subclasses that can make the creation of certain operations much easier. ### 6.3.1 General Case Example: PipeOpCopy A very simple yet useful PipeOp is PipeOpCopy, which takes a single input and creates a variable number of output channels, all of which receive a copy of the input data. It is a simple example that showcases the important steps in defining a custom PipeOp. We will show a simplified version here, PipeOpCopyTwo, that creates exactly two copies of its input data. The following figure visualizes how our PipeOp is situated in the Pipeline and the significant in- and outputs. #### 6.3.1.1 First Steps: Inheriting from PipeOp The first part of creating a custom PipeOp is inheriting from PipeOp. We make a mental note that we need to implement a .train() and a .predict() function, and that we probably want to have an initialize() as well: PipeOpCopyTwo = R6::R6Class("PipeOpCopyTwo", inherit = mlr3pipelines::PipeOp, public = list( initialize = function(id = "copy.two") { .... }, ), private == list( .train = function(inputs) { .... }, .predict = function(inputs) { .... } ) ) Note, that private methods, e.g. .train and .predict etc are prefixed with a .. #### 6.3.1.2 Channel Definitions We need to tell the PipeOp the layout of its channels: How many there are, what their names are going to be, and what types are acceptable. This is done on initialization of the PipeOp (using a super$initialize call) by giving the input and output data.table objects. These must have three columns: a "name" column giving the names of input and output channels, and a "train" and "predict" column naming the class of objects we expect during training and prediction as input / output. A special value for these classes is "*", which indicates that any class will be accepted; our simple copy operator accepts any kind of input, so this will be useful. We have only one input, but two output channels.

By convention, we name a single channel "input" or "output", and a group of channels ["input1", "input2", …], unless there is a reason to give specific different names. Therefore, our input data.table will have a single row <"input", "*", "*">, and our output table will have two rows, <"output1", "*", "*"> and <"output2", "*", "*">.

All of this is given to the PipeOp creator. Our initialize() will thus look as follows:

    initialize = function(id = "copy.two") {
input = data.table::data.table(name = "input", train = "*", predict = "*")
# the following will create two rows and automatically fill the train
# and predict cols with "*"
output = data.table::data.table(
name = c("output1", "output2"),
train = "*", predict = "*"
)
super$initialize(id, input = input, output = output ) } #### 6.3.1.3 Train and Predict Both .train() and .predict() will receive a list as input and must give a list in return. According to our input and output definitions, we will always get a list with a single element as input, and will need to return a list with two elements. Because all we want to do is create two copies, we will just create the copies using c(inputs, inputs). Two things to consider: • The .train() function must always modify the self$state variable to something that is not NULL or NO_OP. This is because the $state slot is used as a signal that PipeOp has been trained on data, even if the state itself is not important to the PipeOp (as in our case). Therefore, our .train() will set self$state = list().

• It is not necessary to “clone” our input or make deep copies, because we don’t modify the data. However, if we were changing a reference-passed object, for example by changing data in a Task, we would have to make a deep copy first. This is because a PipeOp may never modify its input object by reference.

Our .train() and .predict() functions are now:

.train = function(inputs) {
self$state = list() c(inputs, inputs) } .predict = function(inputs) { c(inputs, inputs) } #### 6.3.1.4 Putting it Together The whole definition thus becomes PipeOpCopyTwo = R6::R6Class("PipeOpCopyTwo", inherit = mlr3pipelines::PipeOp, public = list( initialize = function(id = "copy.two") { super$initialize(id,
input = data.table::data.table(name = "input", train = "*", predict = "*"),
output = data.table::data.table(name = c("output1", "output2"),
train = "*", predict = "*")
)
}
),
private = list(
.train = function(inputs) {
self$state = list() c(inputs, inputs) }, .predict = function(inputs) { c(inputs, inputs) } ) ) We can create an instance of our PipeOp, put it in a graph, and see what happens when we train it on something: library("mlr3pipelines") poct = PipeOpCopyTwo$new()
gr = Graph$new() gr$add_pipeop(poct)

print(gr)
## Graph with 1 PipeOps:
##        ID         State sccssors prdcssors
##  copy.two <<UNTRAINED>>
result = gr$train(task) str(result) ## List of 2 ##$ copy.two.output1:Classes 'TaskClassif', 'TaskSupervised', 'Task', 'R6' <TaskClassif:iris>
##  $copy.two.output2:Classes 'TaskClassif', 'TaskSupervised', 'Task', 'R6' <TaskClassif:iris> ### 6.3.2 Special Case: Preprocessing Many PipeOps perform an operation on exactly one Task, and return exactly one Task. They may even not care about the “Target” / “Outcome” variable of that task, and only do some modification of some input data. However, it is usually important to them that the Task on which they perform prediction has the same data columns as the Task on which they train. For these cases, the auxiliary base class PipeOpTaskPreproc exists. It inherits from PipeOp itself, and other PipeOps should use it if they fall in the kind of use-case named above. When inheriting from PipeOpTaskPreproc, one must either implement the private methods .train_task() and .predict_task(), or the methods .train_dt(), .predict_dt(), depending on whether wants to operate on a Task object or on its data as data.tables. In the second case, one can optionally also overload the .select_cols() method, which chooses which of the incoming Task’s features are given to the .train_dt() / .predict_dt() functions. The following will show two examples: PipeOpDropNA, which removes a Task’s rows with missing values during training (and implements .train_task() and .predict_task()), and PipeOpScale, which scales a Task’s numeric columns (and implements .train_dt(), .predict_dt(), and .select_cols()). #### 6.3.2.1 Example: PipeOpDropNA Dropping rows with missing values may be important when training a model that can not handle them. Because mlr3 Tasks only contain a view to the underlying data, it is not necessary to modify data to remove rows with missing values. Instead, the rows can be removed using the Task’s $filter method, which modifies the Task in-place. This is done in the private method .train_task(). We take care that we also set the $state slot to signal that the PipeOp was trained. The private method .predict_task() does not need to do anything; removing missing values during prediction is not as useful, since learners that cannot handle them will just ignore the respective rows. Furthermore, mlr3 expects a Learner to always return just as many predictions as it was given input rows, so a PipeOp that removes Task rows during training can not be used inside a GraphLearner. When we inherit from PipeOpTaskPreproc, it sets the input and output data.tables for us to only accept a single Task. The only thing we do during initialize() is therefore to set an id (which can optionally be changed by the user). The complete PipeOpDropNA can therefore be written as follows. Note that it inherits from PipeOpTaskPreproc, unlike the PipeOpCopyTwo example from above: PipeOpDropNA = R6::R6Class("PipeOpDropNA", inherit = mlr3pipelines::PipeOpTaskPreproc, public = list( initialize = function(id = "drop.na") { super$initialize(id)
}
),

private = list(
self$state = list() featuredata = task$data(cols = task$feature_names) exclude = apply(is.na(featuredata), 1, any) task$filter(task$row_ids[!exclude]) }, .predict_task = function(task) { # nothing to be done task } ) ) To test this PipeOp, we create a small task with missing values: smalliris = iris[(1:5) * 30, ] smalliris[1, 1] = NA smalliris[2, 2] = NA sitask = as_task_classif(smalliris, target = "Species") print(sitask$data())
##       Species Petal.Length Petal.Width Sepal.Length Sepal.Width
## 1:     setosa          1.6         0.2           NA         3.2
## 2: versicolor          3.9         1.4          5.2          NA
## 3: versicolor          4.0         1.3          5.5         2.5
## 4:  virginica          5.0         1.5          6.0         2.2
## 5:  virginica          5.1         1.8          5.9         3.0

We test this by feeding it to a new Graph that uses PipeOpDropNA.

gr = Graph$new() gr$add_pipeop(PipeOpDropNA$new()) filtered_task = gr$train(sitask)[[1]]
print(filtered_task$data()) ## Species Petal.Length Petal.Width Sepal.Length Sepal.Width ## 1: versicolor 4.0 1.3 5.5 2.5 ## 2: virginica 5.0 1.5 6.0 2.2 ## 3: virginica 5.1 1.8 5.9 3.0 #### 6.3.2.2 Example: PipeOpScaleAlways An often-applied preprocessing step is to simply center and/or scale the data to mean $$0$$ and standard deviation $$1$$. This fits the PipeOpTaskPreproc pattern quite well. Because it always replaces all columns that it operates on, and does not require any information about the task’s target, it only needs to overload the .train_dt() and .predict_dt() functions. This saves some boilerplate-code from getting the correct feature columns out of the task, and replacing them after modification. Because scaling only makes sense on numeric features, we want to instruct PipeOpTaskPreproc to give us only these numeric columns. We do this by overloading the .select_cols() function: It is called by the class to determine which columns to pass to .train_dt() and .predict_dt(). Its input is the Task that is being transformed, and it should return a character vector of all features to work with. When it is not overloaded, it uses all columns; instead, we will set it to only give us numeric columns. Because the levels() of the data table given to .train_dt() and .predict_dt() may be different from the task’s levels, these functions must also take a levels argument that is a named list of column names indicating their levels. When working with numeric data, this argument can be ignored, but it should be used instead of levels(dt[[column]]) for factorial or character columns. This is the first PipeOp where we will be using the $state slot for something useful: We save the centering offset and scaling coefficient and use it in $.predict()! For simplicity, we are not using hyperparameters and will always scale and center all data. Compare this PipeOpScaleAlways operator to the one defined inside the mlr3pipelines package, PipeOpScale. PipeOpScaleAlways = R6::R6Class("PipeOpScaleAlways", inherit = mlr3pipelines::PipeOpTaskPreproc, public = list( initialize = function(id = "scale.always") { super$initialize(id = id)
}
),

private = list(
task$feature_types[type == "numeric", id] }, .train_dt = function(dt, levels, target) { sc = scale(as.matrix(dt)) self$state = list(
center = attr(sc, "scaled:center"),
scale = attr(sc, "scaled:scale")
)
sc
},

.predict_dt = function(dt, levels) {
t((t(dt) - self$state$center) / self$state$scale)
}
)
)

(Note for the observant: If you check PipeOpScale.R from the mlr3pipelines package, you will notice that is uses “get("type")” and “get("id")” instead of “type” and “id,” because the static code checker on CRAN would otherwise complain about references to undefined variables. This is a “problem” with data.table and not exclusive to mlr3pipelines.)

We can, again, create a new Graph that uses this PipeOp to test it. Compare the resulting data to the original “iris” Task data printed at the beginning:

gr = Graph$new() gr$add_pipeop(PipeOpScaleAlways$new()) result = gr$train(task)

result[[1]]$data() ## Species Petal.Length Petal.Width Sepal.Length Sepal.Width ## 1: setosa -1.3358 -1.3111 -0.89767 1.01560 ## 2: setosa -1.3358 -1.3111 -1.13920 -0.13154 ## 3: setosa -1.3924 -1.3111 -1.38073 0.32732 ## 4: setosa -1.2791 -1.3111 -1.50149 0.09789 ## 5: setosa -1.3358 -1.3111 -1.01844 1.24503 ## --- ## 146: virginica 0.8169 1.4440 1.03454 -0.13154 ## 147: virginica 0.7036 0.9192 0.55149 -1.27868 ## 148: virginica 0.8169 1.0504 0.79301 -0.13154 ## 149: virginica 0.9302 1.4440 0.43072 0.78617 ## 150: virginica 0.7602 0.7880 0.06843 -0.13154 ### 6.3.3 Special Case: Preprocessing with Simple Train It is possible to make even further simplifications for many PipeOps that perform mostly the same operation during training and prediction. The point of Task preprocessing is often to modify the training data in mostly the same way as prediction data (but in a way that may depend on training data). Consider constant feature removal, for example: The goal is to remove features that have no variance, or only a single factor level. However, what features get removed must be decided during training, and may only depend on training data. Furthermore, the actual process of removing features is the same during training and prediction. A simplification to make is therefore to have a private method .get_state(task) which sets the $state slot during training, and a private method .transform(task), which gets called both during training and prediction. This is done in the PipeOpTaskPreprocSimple class. Just like PipeOpTaskPreproc, one can inherit from this and overload these functions to get a PipeOp that performs preprocessing with very little boilerplate code.

Just like PipeOpTaskPreproc, PipeOpTaskPreprocSimple offers the possibility to instead overload the .get_state_dt(dt, levels) and .transform_dt(dt, levels) methods (and optionally, again, the .select_cols(task) function) to operate on data.table feature data instead of the whole Task.

Even some methods that do not use PipeOpTaskPreprocSimple could work in a similar way: The PipeOpScaleAlways example from above will be shown to also work with this paradigm.

#### 6.3.3.1 Example: PipeOpDropConst

A typical example of a preprocessing operation that does almost the same operation during training and prediction is an operation that drops features depending on a criterion that is evaluated during training. One simple example of this is dropping constant features. Because the mlr3 Task class offers a flexible view on underlying data, it is most efficient to drop columns from the task directly using its $select() function, so the .get_state_dt(dt, levels) / .transform_dt(dt, levels) functions will not get used; instead we overload the .get_state(task) and .transform(task) methods. The .get_state() function’s result is saved to the $state slot, so we want to return something that is useful for dropping features. We choose to save the names of all the columns that have nonzero variance. For brevity, we use length(unique(column)) > 1 to check whether more than one distinct value is present; a more sophisticated version could have a tolerance parameter for numeric values that are very close to each other.

The .transform() method is evaluated both during training and prediction, and can rely on the $state slot being present. All it does here is call the Task$select function with the columns we chose to keep.

The full PipeOp could be written as follows:

PipeOpDropConst = R6::R6Class("PipeOpDropConst",
public = list(
initialize = function(id = "drop.const") {
super$initialize(id = id) } ), private = list( .get_state = function(task) { data = task$data(cols = task$feature_names) nonconst = sapply(data, function(column) length(unique(column)) > 1) list(cnames = colnames(data)[nonconst]) }, .transform = function(task) { task$select(self$state$cnames)
}
)
)

This can be tested using the first five rows of the “Iris” Task, for which one feature ("Petal.Width") is constant:

irishead = task$clone()$filter(1:5)
irishead$data() ## Species Petal.Length Petal.Width Sepal.Length Sepal.Width ## 1: setosa 1.4 0.2 5.1 3.5 ## 2: setosa 1.4 0.2 4.9 3.0 ## 3: setosa 1.3 0.2 4.7 3.2 ## 4: setosa 1.5 0.2 4.6 3.1 ## 5: setosa 1.4 0.2 5.0 3.6 gr = Graph$new()$add_pipeop(PipeOpDropConst$new())
dropped_task = gr$train(irishead)[[1]] dropped_task$data()
##    Species Petal.Length Sepal.Length Sepal.Width
## 1:  setosa          1.4          5.1         3.5
## 2:  setosa          1.4          4.9         3.0
## 3:  setosa          1.3          4.7         3.2
## 4:  setosa          1.5          4.6         3.1
## 5:  setosa          1.4          5.0         3.6

We can also see that the $state was correctly set. Calling $.predict() with this graph, even with different data (the whole Iris Task!) will still drop the "Petal.Width" column, as it should.

gr$pipeops$drop.const$state ##$cnames
## [1] "Petal.Length" "Sepal.Length" "Sepal.Width"
##
## $affected_cols ## [1] "Petal.Length" "Petal.Width" "Sepal.Length" "Sepal.Width" ## ##$intasklayout
##              id    type
## 1: Petal.Length numeric
## 2:  Petal.Width numeric
## 3: Sepal.Length numeric
## 4:  Sepal.Width numeric
##
## $outtasklayout ## id type ## 1: Petal.Length numeric ## 2: Sepal.Length numeric ## 3: Sepal.Width numeric ## ##$outtaskshell
## Empty data.table (0 rows and 4 cols): Species,Petal.Length,Sepal.Length,Sepal.Width
dropped_predict = gr$predict(task)[[1]] dropped_predict$data()
##        Species Petal.Length Sepal.Length Sepal.Width
##   1:    setosa          1.4          5.1         3.5
##   2:    setosa          1.4          4.9         3.0
##   3:    setosa          1.3          4.7         3.2
##   4:    setosa          1.5          4.6         3.1
##   5:    setosa          1.4          5.0         3.6
##  ---
## 146: virginica          5.2          6.7         3.0
## 147: virginica          5.0          6.3         2.5
## 148: virginica          5.2          6.5         3.0
## 149: virginica          5.4          6.2         3.4
## 150: virginica          5.1          5.9         3.0

#### 6.3.3.2 Example: PipeOpScaleAlwaysSimple

This example will show how a PipeOpTaskPreprocSimple can be used when only working on feature data in form of a data.table. Instead of calling the scale() function, the center and scale values are calculated directly and saved to the $state slot. The .transform_dt() function will then perform the same operation during both training and prediction: subtract the center and divide by the scale value. As in the PipeOpScaleAlways example above, we use .select_cols() so that we only work on numeric columns. PipeOpScaleAlwaysSimple = R6::R6Class("PipeOpScaleAlwaysSimple", inherit = mlr3pipelines::PipeOpTaskPreprocSimple, public = list( initialize = function(id = "scale.always.simple") { super$initialize(id = id)
}
),

private = list(
task$feature_types[type == "numeric", id] }, .get_state_dt = function(dt, levels, target) { list( center = sapply(dt, mean), scale = sapply(dt, sd) ) }, .transform_dt = function(dt, levels) { t((t(dt) - self$state$center) / self$state$scale) } ) ) We can compare this PipeOp to the one above to show that it behaves the same. gr = Graph$new()$add_pipeop(PipeOpScaleAlways$new())
result_posa = gr$train(task)[[1]] gr = Graph$new()$add_pipeop(PipeOpScaleAlwaysSimple$new())
result_posa_simple = gr$train(task)[[1]] result_posa$data()
##        Species Petal.Length Petal.Width Sepal.Length Sepal.Width
##   1:    setosa      -1.3358     -1.3111     -0.89767     1.01560
##   2:    setosa      -1.3358     -1.3111     -1.13920    -0.13154
##   3:    setosa      -1.3924     -1.3111     -1.38073     0.32732
##   4:    setosa      -1.2791     -1.3111     -1.50149     0.09789
##   5:    setosa      -1.3358     -1.3111     -1.01844     1.24503
##  ---
## 146: virginica       0.8169      1.4440      1.03454    -0.13154
## 147: virginica       0.7036      0.9192      0.55149    -1.27868
## 148: virginica       0.8169      1.0504      0.79301    -0.13154
## 149: virginica       0.9302      1.4440      0.43072     0.78617
## 150: virginica       0.7602      0.7880      0.06843    -0.13154
result_posa_simple$data() ## Species Petal.Length Petal.Width Sepal.Length Sepal.Width ## 1: setosa -1.3358 -1.3111 -0.89767 1.01560 ## 2: setosa -1.3358 -1.3111 -1.13920 -0.13154 ## 3: setosa -1.3924 -1.3111 -1.38073 0.32732 ## 4: setosa -1.2791 -1.3111 -1.50149 0.09789 ## 5: setosa -1.3358 -1.3111 -1.01844 1.24503 ## --- ## 146: virginica 0.8169 1.4440 1.03454 -0.13154 ## 147: virginica 0.7036 0.9192 0.55149 -1.27868 ## 148: virginica 0.8169 1.0504 0.79301 -0.13154 ## 149: virginica 0.9302 1.4440 0.43072 0.78617 ## 150: virginica 0.7602 0.7880 0.06843 -0.13154 ### 6.3.4 Hyperparameters mlr3pipelines uses the paradox package to define parameter spaces for PipeOps. Parameters for PipeOps can modify their behavior in certain ways, e.g. switch centering or scaling off in the PipeOpScale operator. The unified interface makes it possible to have parameters for whole Graphs that modify the individual PipeOp’s behavior. The Graphs, when encapsulated in GraphLearners, can even be tuned using the tuning functionality in mlr3tuning. Hyperparameters are declared during initialization, when calling the PipeOp’s $initialize() function, by giving a param_set argument. The param_set must be a ParamSet from the paradox package; see the tuning chapter or the in-depth paradox chapter for more information on how to define parameter spaces. After construction, the ParamSet can be accessed through the $param_set slot. While it is possible to modify this ParamSet, using e.g. the $add() and $add_dep() functions, after adding it to the PipeOp, it is strongly advised against. Hyperparameters can be set and queried through the $values slot. When setting hyperparameters, they are automatically checked to satisfy all conditions set by the $param_set, so it is not necessary to type check them. Be aware that it is always possible to remove hyperparameter values. When a PipeOp is initialized, it usually does not have any parameter values—$values takes the value list(). It is possible to set initial parameter values in the $initialize() constructor; this must be done after the super$initialize() call where the corresponding ParamSet must be supplied. This is because setting $values checks against the current $param_set, which would fail if the $param_set was not set yet. When using an underlying library function (the scale function in PipeOpScale, say), then there is usually a “default” behaviour of that function when a parameter is not given. It is good practice to use this default behaviour whenever a parameter is not set (or when it was removed). This can easily be done when using the mlr3misc library’s mlr3misc::invoke() function, which has functionality similar to do.call(). #### 6.3.4.1 Hyperparameter Example: PipeOpScale How to use hyperparameters can best be shown through the example of PipeOpScale, which is very similar to the example above, PipeOpScaleAlways. The difference is made by the presence of hyperparameters. PipeOpScale constructs a ParamSet in its $initialize function and passes this on to the super$initialize function: PipeOpScale$public_methods$initialize ## function (id = "scale", param_vals = list()) ## .__PipeOpScale__initialize(self = self, private = private, super = super, ## id = id, param_vals = param_vals) ## <environment: namespace:mlr3pipelines> The user has access to this and can set and get parameters. Types are automatically checked: pss = po("scale") print(pss$param_set)
## <ParamSet:scale>
##                id    class lower upper nlevels        default value
## 1:         center ParamLgl    NA    NA       2           TRUE
## 2:          scale ParamLgl    NA    NA       2           TRUE
## 3:         robust ParamLgl    NA    NA       2 <NoDefault[3]> FALSE
## 4: affect_columns ParamUty    NA    NA     Inf  <Selector[1]>
pss$param_set$values$center = FALSE print(pss$param_set$values) ##$robust
## [1] FALSE
##
## $center ## [1] FALSE pss$param_set$values$scale = "TRUE"  # bad input is checked!
## Error in self$assert(xs): Assertion on 'xs' failed: scale: Must be of type 'logical flag', not 'character'. How PipeOpScale handles its parameters can be seen in its $.train_dt method: It gets the relevant parameters from its $values slot and uses them in the mlr3misc::invoke() call. This has the advantage over calling scale() directly that if a parameter is not given, its default value from the scale() function will be used. PipeOpScale$private_methods$.train_dt ## function (dt, levels, target) ## .__PipeOpScale__.train_dt(self = self, private = private, super = super, ## dt = dt, levels = levels, target = target) ## <environment: namespace:mlr3pipelines> Another change that is necessary compared to PipeOpScaleAlways is that the attributes "scaled:scale" and "scaled:center" are not always present, depending on parameters, and possibly need to be set to default values $$1$$ or $$0$$, respectively. It is now even possible (if a bit pointless) to call PipeOpScale with both scale and center set to FALSE, which returns the original dataset, unchanged. pss$param_set$values$scale = FALSE
pss$param_set$values$center = FALSE gr = Graph$new()
gr$add_pipeop(pss) result = gr$train(task)

result[[1]]\$data()
##        Species Petal.Length Petal.Width Sepal.Length Sepal.Width
##   1:    setosa          1.4         0.2          5.1         3.5
##   2:    setosa          1.4         0.2          4.9         3.0
##   3:    setosa          1.3         0.2          4.7         3.2
##   4:    setosa          1.5         0.2          4.6         3.1
##   5:    setosa          1.4         0.2          5.0         3.6
##  ---
## 146: virginica          5.2         2.3          6.7         3.0
## 147: virginica          5.0         1.9          6.3         2.5
## 148: virginica          5.2         2.0          6.5         3.0
## 149: virginica          5.4         2.3          6.2         3.4
## 150: virginica          5.1         1.8          5.9         3.0